Cluster Test and Verification

Track 1: Validation strategy and test layering

You are expected to run tests in a disciplined order from single-node sanity to multi-node fabric confidence.

• - Start with single-node validation before multi-node collectives.
• - Use progressively stronger tests: readiness, performance, and burn-in stability.
• - Define clear pass/fail thresholds per test category.

Drill: Design a layered validation plan showing test order, thresholds, and stop conditions.

Track 2: HPL execution and burn-in

HPL is explicitly listed in exam objectives for verification and burn-in.

• - Differentiate quick HPL functional runs from long-duration burn-in runs.
• - Capture not just peak output but stability and repeatability over time.
• - Treat thermal/power stability as part of HPL result interpretation.

Drill: Run an HPL baseline protocol and document acceptance criteria for burn-in promotion.

Track 3: NCCL communication validation

NCCL tests verify east-west bandwidth, collectives behavior, and NVLink/NVSwitch paths.

• - Run single-node NCCL first, then expand to multi-node paths.
• - Use NCCL diagnostics and debug settings when behavior diverges from baseline.
• - Correlate NCCL outcomes with topology, cabling, and firmware state.

Drill: Execute a two-stage NCCL test sequence (single-node then multi-node) and explain result deltas.

Track 4: ClusterKit multifaceted node assessment

Blueprint explicitly calls out ClusterKit for broad node and communication assessment.

• - ClusterKit combines latency, bandwidth, collective, and stress-style checks.
• - ClusterKit execution should align with scheduler or passwordless orchestration prerequisites.
• - Use ClusterKit findings to prioritize deeper NCCL/HPL investigations.

Drill: Run a minimal ClusterKit scenario and produce a triage list from its findings.

Track 5: Firmware, cabling, and transceiver verification

Fabric quality issues can invalidate benchmark conclusions if not verified first.

• - Confirm firmware/software alignment on switches and BlueField components.
• - Validate cable routes and transceiver compatibility to prevent hidden link issues.
• - Treat signal-quality checks as prerequisites for interpreting bandwidth anomalies.

Drill: Create a pre-benchmark fabric checklist covering cable, transceiver, switch, and BlueField state.

Track 6: Storage verification in cluster readiness

Storage bottlenecks can mimic compute/network instability in end-to-end validation.

• - Include storage checks in readiness and burn-in flows, not only in post-failure triage.
• - Correlate storage throughput baselines with workload data path expectations.
• - Escalate storage anomalies before interpreting model or benchmark regressions.

Drill: Pair one storage test with one NCCL/HPL run and explain cross-layer interpretation.

Exam Decision Hierarchy

Prioritize decisions in this order: safety and hardware integrity, baseline consistency, controlled validation, then optimization.

• - If integrity checks fail, stop optimization and remediate first.
• - Compare against known-good baseline before changing multiple variables.
• - Document rationale for each decision to support incident replay.

Operational Evidence Standard

Treat every key action as evidence-producing: command, output, timestamp, and expected vs observed behavior.

• - Evidence should be reproducible by another engineer.
• - Use stable command templates for repeated environments.
• - Keep concise but complete validation artifacts for exam-style reasoning.

Validation ladder design

Reliable cluster verification uses a strict progression from local sanity to distributed stress and long-duration burn-in.

• - Start single-node to reduce fault-space before multi-node collectives.
• - Separate quick readiness checks from prolonged stability tests.
• - Use explicit promotion gates between each layer.

HPL and NCCL interpretation discipline

Benchmark numbers are only useful when interpreted with topology, firmware, and thermal context.

• - A high peak score does not guarantee stable production operation.
• - NCCL anomalies often require fabric and firmware correlation.
• - Burn-in success requires duration + error-free behavior, not one run.

Integrated readiness across compute, fabric, and storage

Cluster readiness is end-to-end; storage and networking signals must be evaluated alongside compute metrics.

• - Include storage tests in validation plan, not only post-incident.
• - Run cross-layer diagnostics when regressions appear.
• - Preserve test artifacts for trend comparison across maintenance cycles.

Scenario A: Multi-node NCCL bandwidth lower than baseline

After infrastructure changes, multi-node all_reduce performance drops sharply while single-node tests remain stable.

[Node A GPUs] -- NVLink/NVSwitch -- [Node A]
      |                                |
  InfiniBand/Ethernet Fabric (E/W) between nodes
      |                                |
[Node B GPUs] -- NVLink/NVSwitch -- [Node B]

all_reduce_perf -b 8 -e 1G -f 2 -g 8

# size  count  type  redop  root  time(us)  algbw(GB/s)\n...\n1073741824 ... 356.2

Scenario B: HPL peak looks good but burn-in fails intermittently

Quick HPL run meets target, but long-duration burn-in reports periodic failures.

[Validation Pipeline]
  -> [HPL Functional] -> [NCCL Functional] -> [Burn-in Window]
                                  |
                             [Thermal/Power Telemetry]

nvidia-smi --query-gpu=timestamp,temperature.gpu,power.draw,utilization.gpu --format=csv -l 30

2026/02/17 10:10:00, 62 C, 287 W, 96 %\n2026/02/17 10:10:30, 63 C, 291 W, 97 %

Single-node to multi-node NCCL runbook

Apply this sequence to validate communication path progressively and avoid ambiguous multi-node failures.

all_reduce_perf -b 8 -e 512M -f 2 -g 8

Single-node NCCL baseline captured with expected intra-node bandwidth.

mpirun -np 16 -H node1:8,node2:8 all_reduce_perf -b 8 -e 1G -f 2 -g 1

Cross-node bandwidth and latency metrics generated for comparison.

• - Keep message-size and process mapping consistent between baseline and comparison runs.
• - Pair results with fabric and firmware state snapshot.

HPL + burn-in runbook

Use a two-phase pattern: functional performance check, then long-duration stability check.

./hpl.sh --dat ./HPL.dat

HPL run complete: baseline GFLOPS captured.

for i in {1..12}; do ./hpl.sh --dat ./HPL.dat; sleep 300; done

12-cycle burn-in complete with no fatal errors.

• - Collect thermal/power telemetry in parallel.
• - Fail promotion if any cycle produces unexplained node-level instability.

Single-node healthy, multi-node NCCL degraded

Burn-in instability despite good functional benchmark score

Storage bottleneck misread as compute/network issue

Walkthrough: Full cluster verification ladder

Execute complete readiness ladder from single-node checks to burn-in and storage validation.

1. Run single-node stress and NCCL functional checks.

   all_reduce_perf -b 8 -e 512M -f 2 -g 8

   Expected: Single-node communication baseline captured with no errors.

2. Execute HPL functional baseline.

   ./hpl.sh --dat ./HPL.dat

   Expected: Baseline output recorded and within expected range.

3. Run multi-node NCCL E/W bandwidth validation.

   mpirun -np 16 -H node1:8,node2:8 all_reduce_perf -b 8 -e 1G -f 2 -g 1

   Expected: Cross-node bandwidth within policy threshold.

4. Start NCCL/HPL burn-in cycles with telemetry capture.

   Expected: No repeated instability signatures during burn-in window.

5. Run storage validation and correlate with benchmark artifacts.

   fio --name=clusterread --directory=/mnt/ai-data --rw=read --bs=1M --size=16G --numjobs=8

   Expected: Storage throughput and latency meet readiness policy.

Lab A: Single-node to multi-node validation ladder

Execute validation in progressive layers and capture confidence evidence.

• - Run single-node stress/HPL/NCCL baseline.
• - Promote to multi-node NCCL only after pass criteria.
• - Document where failures first appear in the ladder.

all_reduce_perf -b 8 -e 512M -f 2 -g 8

Single-node NCCL baseline captured with expected intra-node bandwidth.

Lab B: ClusterKit + NCCL triangulation

Use multiple tools to isolate communication issues.

• - Run ClusterKit assessment and capture anomalies.
• - Run targeted NCCL tests for confirmation.
• - Map anomalies to topology/cabling/firmware candidates.

mpirun -np 16 -H node1:8,node2:8 all_reduce_perf -b 8 -e 1G -f 2 -g 1

Cross-node bandwidth and latency metrics generated for comparison.

Lab C: Burn-in reliability workflow

Validate cluster stability under prolonged load.

• - Execute NCCL and HPL burn-in runs with monitoring.
• - Track thermal and error-state behavior over time.
• - Define go/no-go criteria for production entry.

./hpl.sh --dat ./HPL.dat

HPL run complete: baseline GFLOPS captured.

Lab D: End-to-end data-path verification

Validate compute, communication, and storage readiness together.

• - Run one NeMo or representative workload stress pass.
• - Pair with storage throughput/latency checks.
• - Produce integrated readiness statement with evidence.

for i in {1..12}; do ./hpl.sh --dat ./HPL.dat; sleep 300; done

12-cycle burn-in complete with no fatal errors.