Troubleshoot and Optimize

Track 1: Fault-domain triage framework

Fast isolation across hardware, firmware, driver, and fabric layers is critical in production incidents.

• - Start with symptom classification: compute, thermal, power, network, or storage.
• - Use deterministic triage order to avoid random command-driven debugging.
• - Correlate telemetry across host and fabric before replacing hardware.

Drill: Build an incident triage flowchart mapping symptom patterns to likely fault domains.

Track 2: GPU and host health diagnostics

Exam tasks include identifying faulty GPUs/cards/power components, not only software errors.

• - Use standardized GPU telemetry and diagnostics to detect thermal, ECC, and stability issues.
• - Differentiate transient software errors from persistent hardware fault signatures.
• - Capture pre-replacement evidence to support RMA or maintenance decisions.

Drill: Run a structured GPU diagnostic pass and produce a replace vs observe decision memo.

Track 3: Component replacement workflow discipline

Safe replacement procedures reduce repeat incidents and post-maintenance regressions.

• - Define replacement prerequisites: evidence, approvals, spares, rollback and validation plan.
• - Re-validate firmware/driver alignment after component swaps.
• - Use post-replacement burn-in checks before returning node to service.

Drill: Write a component replacement SOP with evidence capture, execution controls, and exit criteria.

Track 4: CPU platform optimization (AMD/Intel server context)

Blueprint includes optimization workflows on mixed server CPU platforms.

• - Treat BIOS/NUMA/power-profile tuning as workload-dependent, not one-size-fits-all.
• - Benchmark after each tuning change to avoid placebo or regression effects.
• - Validate CPU and memory policy interaction with GPU pipeline performance.

Drill: Create a CPU tuning experiment table with one change per run and measured impact.

Track 5: Storage path optimization

Storage bottlenecks frequently appear as training or inference slowdowns.

• - Separate storage latency issues from network or compute issues during triage.
• - Use storage benchmarks and telemetry for evidence-driven tuning.
• - Evaluate direct storage-to-GPU paths where supported.

Drill: Run storage baseline tests and produce a prioritized tuning backlog.

Track 6: Verification after remediation

Troubleshooting is incomplete without proving the fix holds under realistic load.

• - Require post-fix validation with representative workload and duration.
• - Track recurrence indicators and define rollback/escalation thresholds.
• - Update runbooks and knowledge base from incident outcomes.

Drill: Create a post-remediation validation template with recurrence watchpoints.

Exam Decision Hierarchy

Prioritize decisions in this order: safety and hardware integrity, baseline consistency, controlled validation, then optimization.

• - If integrity checks fail, stop optimization and remediate first.
• - Compare against known-good baseline before changing multiple variables.
• - Document rationale for each decision to support incident replay.

Operational Evidence Standard

Treat every key action as evidence-producing: command, output, timestamp, and expected vs observed behavior.

• - Evidence should be reproducible by another engineer.
• - Use stable command templates for repeated environments.
• - Keep concise but complete validation artifacts for exam-style reasoning.

Evidence-first triage model

Reliable troubleshooting starts with symptom-to-domain mapping before remediation action.

• - Classify incident by compute, thermal/power, network, storage, or mixed signals.
• - Use smallest set of high-signal diagnostics first.
• - Preserve evidence before component replacement.

Replace vs tune vs rollback decisions

Operational excellence requires choosing the right remediation path based on confidence and blast radius.

• - Replace components only when evidence supports hardware fault signatures.
• - Use tuning when behavior is stable but below expected performance target.
• - Rollback after high-risk changes when confidence in root cause is low.

Post-fix verification and recurrence control

A fix is incomplete until it holds under representative workload duration and no recurrence indicators remain.

• - Validate with the same workload class that exposed the issue.
• - Track recurrence watchpoints for at least one stability window.
• - Update runbooks and alerts with discovered failure signatures.

Scenario A: Intermittent GPU Xid errors under load

A node passes smoke tests but emits intermittent Xid-like failures during sustained workload periods.

[Workload Scheduler] -> [GPU Node]
                      |
               [Telemetry + Kernel Logs]
                      |
                [Decision: replace/tune/rollback]

dmesg | rg -i 'xid|nvrm|pcie' | tail -n 30

[NVRM] Xid ...\n[NVRM] Channel exception ... (example)

Scenario B: Throughput regression after platform tuning changes

Performance dropped after combined BIOS and runtime tuning changes on a mixed AMD/Intel fleet.

[Baseline Config] -> [Change Set A+B+C] -> [Regression]
      |                     |
  benchmark pack       unclear causality
      \_________________rollback + one-change retest________________/

lscpu && numactl --hardware

Architecture: x86_64\nNUMA node(s): 2\nnode distances: ...

Fault isolation command runbook

Use this compact set to classify incidents quickly without command sprawl.

nvidia-smi -q -d TEMPERATURE,POWER,UTILIZATION,ECC,PERFORMANCE

Temperature : 61 C\nPower Draw : 286 W\nEcc Mode : Enabled\n...

journalctl -k -n 200 | rg -i 'nvrm|xid|pcie|aer'

kernel: NVRM: Xid ...\nkernel: pcieport ... (example)

all_reduce_perf -b 8 -e 256M -f 2 -g 8

NCCL sanity run complete with measured bandwidth table.

• - Use timestamp alignment across logs and telemetry to build root-cause confidence.
• - Capture command outputs before any replacement action.

Storage and host optimization runbook

Separate host and storage effects when investigating throughput regressions.

fio --name=optcheck --directory=/mnt/ai-data --rw=readwrite --bs=1M --size=8G --numjobs=4 --iodepth=16

READ: bw=8.4GiB/s ...\nWRITE: bw=6.9GiB/s ...

mpstat -P ALL 1 5

Average: CPU %usr %sys %iowait ... (example)

• - Keep workload and dataset constant when comparing tuning outcomes.
• - If storage bottleneck dominates, host tuning alone will not recover target performance.

False hardware replacement due to incomplete diagnosis

Performance tuning causes unstable workloads

Storage bottleneck hidden behind GPU utilization drops

Walkthrough: Detect -> isolate -> remediate -> verify cycle

Execute full incident-response cycle for a realistic infrastructure fault and prove durable recovery.

1. Collect initial high-signal diagnostics and classify fault domain.

   nvidia-smi -q -d TEMPERATURE,POWER,ECC && journalctl -k -n 200

   Expected: Symptoms mapped to a prioritized fault domain.

2. Run focused reproducer for suspected domain.

   all_reduce_perf -b 8 -e 256M -f 2 -g 8

   Expected: Issue is reproduced or ruled out in communication path.

3. Apply least-risk remediation (tune, rollback, or replace).

   Expected: Remediation executed with change record and rollback plan.

4. Perform post-fix workload validation window.

   nvidia-smi --query-gpu=timestamp,utilization.gpu,temperature.gpu,power.draw --format=csv -l 30

   Expected: No recurrence indicators during defined validation duration.

5. Close incident with updated runbook entry.

   Expected: Knowledge base updated with signature, fix, and prevention controls.

Lab A: Fault-domain isolation

Identify root cause domain using minimal but high-signal diagnostics.

• - Collect initial symptoms and classify likely domain.
• - Run focused diagnostics and avoid blind command sweeps.
• - Document root-cause confidence and alternatives.

nvidia-smi -q -d TEMPERATURE,POWER,UTILIZATION,ECC,PERFORMANCE

Temperature : 61 C\nPower Draw : 286 W\nEcc Mode : Enabled\n...

Lab B: Replace-and-validate workflow

Execute safe component replacement with proof of fix.

• - Capture pre-replacement fault evidence.
• - Perform controlled replacement and baseline restore.
• - Run post-replacement validation under representative load.

journalctl -k -n 200 | rg -i 'nvrm|xid|pcie|aer'

kernel: NVRM: Xid ...\nkernel: pcieport ... (example)

Lab C: CPU and host-level optimization

Measure performance impact of one-at-a-time platform tuning changes.

• - Select one AMD/Intel host tuning candidate per run.
• - Collect before/after metrics with constant workload.
• - Keep only changes with reproducible improvement.

all_reduce_perf -b 8 -e 256M -f 2 -g 8

NCCL sanity run complete with measured bandwidth table.

Lab D: Storage tuning and regression guardrails

Improve storage path performance while preventing regressions.

• - Measure baseline throughput/latency and variability.
• - Apply prioritized storage tuning changes.
• - Define regression alarms for post-change monitoring.

fio --name=optcheck --directory=/mnt/ai-data --rw=readwrite --bs=1M --size=8G --numjobs=4 --iodepth=16

READ: bw=8.4GiB/s ...\nWRITE: bw=6.9GiB/s ...