Troubleshooting and Optimization

Track 1: Scheduler and orchestration troubleshooting

Scheduler issues are high-frequency failure points that directly impact workload availability.

• - Use event and status timelines to localize control-plane failures.
• - Differentiate policy misconfiguration from capacity constraints.
• - Validate fixes under representative workload pressure.

Drill: Build a triage sequence for Pending workloads across two scheduler environments.

Track 2: Conversion and workflow fault isolation

Conversion and route failures can appear as runtime instability unless diagnosed systematically.

• - Trace artifact conversion outputs to runtime compatibility checks.
• - Identify stage transitions where workflows break.
• - Recover with minimal blast radius using rollback artifacts.

Drill: Given failing inference output, isolate whether issue is conversion, route, or runtime.

Track 3: Workload-level diagnostics

You must separate infrastructure symptoms from workload misconfiguration quickly.

• - Correlate logs, metrics, and events for one workload timeline.
• - Check resource pressure and runtime errors before tuning.
• - Use repeated validation windows to confirm fix durability.

Drill: Create a one-page workload triage template with evidence checkpoints.

Track 4: Fabric and network diagnostics

Fabric diagnostics are explicitly listed in blueprint and often dominate distributed AI failure modes.

• - Identify whether issue is node-local, path-local, or fabric-wide.
• - Use network and GPU topology evidence to localize bottlenecks.
• - Validate remediation without introducing new policy risks.

Drill: Run a fabric diagnostic flow and classify issue scope in under 12 minutes.

Track 5: AI workload optimization

Optimization is part of domain scope and must be evidence-driven, not anecdotal.

• - Profile bottlenecks before changing configuration.
• - Tune one variable at a time and measure impact.
• - Promote changes only when gains are repeatable.

Drill: Perform one optimization cycle with before/after metrics and rollback criteria.

Exam Decision Hierarchy

Prioritize decisions in this order: safety and hardware integrity, baseline consistency, controlled validation, then optimization.

• - If integrity checks fail, stop optimization and remediate first.
• - Compare against known-good baseline before changing multiple variables.
• - Document rationale for each decision to support incident replay.

Operational Evidence Standard

Treat every key action as evidence-producing: command, output, timestamp, and expected vs observed behavior.

• - Evidence should be reproducible by another engineer.
• - Use stable command templates for repeated environments.
• - Keep concise but complete validation artifacts for exam-style reasoning.

Layered triage model

Use a layered model (scheduler, runtime, workflow, fabric) to reduce diagnosis time.

• - Identify failing layer before changing settings.
• - Use layer-specific commands and validation gates.
• - Escalate only with collected evidence.

Evidence-driven optimization

Optimization should be treated as a controlled experiment with baseline and repeatability checks.

• - Define target metric and expected gain before change.
• - Tune one lever at a time.
• - Reject changes that do not show repeatable benefit.

Incident durability mindset

A resolved incident is not complete until prevention controls and validation runbooks are updated.

• - Capture root cause, remediation, and prevention in one record.
• - Add guardrail checks to prevent recurrence.
• - Validate prevention controls in later maintenance cycles.

Scenario: Distributed job throughput collapse

A distributed training workflow suddenly drops throughput by 30% with no obvious scheduler errors.

Scheduler
   |
Runtime Pods/Jobs
   |
Fabric + Storage
   |
Model Workflow

kubectl get events -A --sort-by=.lastTimestamp

Timeline highlights scheduler/runtime anomalies in incident window.

nvidia-smi topo -m && nvidia-smi dmon -s pucm

Topology map and utilization counters reveal communication or resource pressure signals.

Scenario: Intermittent inference failures after conversion update

A new converted model passes deployment but produces intermittent inference failures under live traffic.

Conversion Artifact
     |
Runtime Endpoint
     |
Workflow Route
     |
Client Calls

Incident triage baseline runbook

Collect high-signal incident evidence across scheduler and runtime layers.

kubectl get events -A --sort-by=.lastTimestamp

Event timeline identifies first visible failure indicators.

kubectl get pods -A -o wide && kubectl get nodes -o wide

Status and placement context support layer-specific diagnosis.

scontrol show job <jobid>

Job detail clarifies resource and scheduling constraints.

• - Capture artifacts before remediation.
• - Tie outputs to precise incident timestamps.

Fabric and optimization runbook

Diagnose communication path issues and validate optimization impact.

iperf3 -c <peer-host> -P 4 -t 20

Throughput profile highlights path performance limits.

nvidia-smi topo -m

Topology relationships support communication bottleneck analysis.

python3 run_workload_benchmark.py --duration 180

Benchmark output provides before/after optimization comparison.

• - Tune one variable and rerun identical benchmark profile.
• - Reject optimization changes that do not hold in repeat run.

Pending/failed workloads without clear root cause

Intermittent latency spikes in distributed workload

Optimization changes improve one workload but degrade others

Walkthrough: End-to-end troubleshooting cycle

Execute layered incident triage from scheduler symptoms to validated remediation outcome.

1. Collect incident timeline.

   kubectl get events -A --sort-by=.lastTimestamp

   Expected: Timeline reveals initial and downstream failure signals.

2. Inspect workload and resource status.

   kubectl get pods -A -o wide && squeue

   Expected: Status output indicates likely failing layer.

3. Run communication/path diagnostics.

   iperf3 -c <peer-host> -P 4 -t 20 && nvidia-smi topo -m

   Expected: Path and topology output clarify communication bottleneck candidates.

4. Apply one remediation and retest.

   python3 run_workload_benchmark.py --duration 180

   Expected: Metrics recover toward baseline within defined threshold.

Walkthrough: Conversion and route failure recovery

Recover from conversion-induced runtime failure with rollback and guardrail update.

1. Validate failing endpoint behavior.

   curl -sS -X POST http://<endpoint>/v1/completions -H 'Content-Type: application/json' -d '{"prompt":"check","max_tokens":8}'

   Expected: Failure reproduces with traceable error signature.

2. Rollback to known-good artifact.

   python3 rollback_model.py --artifact known-good

   Expected: Service returns to stable response behavior.

3. Run guardrail validation test set.

   python3 validate_output_contract.py --suite production-smoke

   Expected: Validation suite passes and blocks known-bad artifact pattern.

Lab A: Pending workload triage

Diagnose and remediate scheduler-level pending workload condition.

• - Collect events and scheduler state.
• - Identify policy versus capacity root cause.
• - Apply targeted fix and verify scheduling recovery.

kubectl get events -A --sort-by=.lastTimestamp

Event timeline identifies first visible failure indicators.

Lab B: Conversion-to-runtime failure drill

Isolate conversion artifact issue from runtime or route errors.

• - Inspect conversion metadata and runtime logs.
• - Run endpoint contract validation tests.
• - Rollback and compare behavior with known-good artifact.

kubectl get pods -A -o wide && kubectl get nodes -o wide

Status and placement context support layer-specific diagnosis.

Lab C: Fabric diagnostics and bottleneck localization

Use network and topology tools to localize communication bottlenecks.

• - Capture topology and interface baseline.
• - Run targeted path/performance probes.
• - Recommend and validate one remediation step.

scontrol show job <jobid>

Job detail clarifies resource and scheduling constraints.

Lab D: Optimization with repeatability check

Run one optimization cycle and verify durability across repeated runs.

• - Establish baseline metrics and SLO thresholds.
• - Apply one optimization variable change.
• - Validate improvement across at least two windows.

iperf3 -c <peer-host> -P 4 -t 20

Throughput profile highlights path performance limits.