System and Server Bring-up

Track 1: Bring-up sequence and validation gates

This domain expects disciplined sequencing from rack state to workload-ready server validation.

• - Use a repeatable bring-up checklist with explicit pass/fail gates for each step.
• - Separate physical installation, firmware baseline, and workload validation into distinct phases.
• - Capture evidence per phase so later failures can be traced quickly.

Drill: Write a one-node bring-up runbook with stage gates and artifacts collected at each gate.

Track 2: AI factory topology and cabling basics

Wrong topology and cabling assumptions create hidden bottlenecks before software even starts.

• - Map workload communication patterns to north-south and east-west traffic needs.
• - Validate cable type, speed class, and transceiver compatibility before cluster tests.
• - Confirm that planned topology supports target NCCL/HPL validation goals.

Drill: Given a sample rack diagram, identify one topology risk and one cable/transceiver risk.

Track 3: BMC, OOB, and platform security baseline

Initial management-plane setup is required for safe operations and remote lifecycle management.

• - Bring up BMC access and out-of-band management before in-band OS workflows.
• - Establish trusted boot and hardware security baseline checks (including TPM state).
• - Document management network segmentation and credentials handling policy.

Drill: Create a secure first-boot checklist for BMC/OOB onboarding and validation.

Track 4: Firmware lifecycle and fault detection

Exam scope explicitly includes firmware upgrade flow (including HGX) and fault detection.

• - Standardize firmware version baseline across compute, switch, and management components.
• - Use vendor-supported tooling for upgrade planning and post-upgrade verification.
• - Treat partial upgrades as a risk and verify firmware/software matrix consistency.

Drill: Plan one firmware update cycle and list pre-checks, execution steps, and rollback conditions.

Track 5: Power, cooling, and hardware validation

Thermal and power misconfiguration causes performance throttling and false-negative diagnostics.

• - Validate power budget against server profile and rack-level limits.
• - Verify cooling and airflow conditions before stress or burn-in tests.
• - Use baseline health telemetry before running performance workloads.

Drill: Build a pre-benchmark readiness checklist covering power, thermals, and fan-state metrics.

Track 6: Third-party storage readiness

Bring-up is incomplete if storage path parameters are undefined or unstable.

• - Define initial storage connectivity and performance expectations before cluster validation.
• - Validate mount and throughput sanity on first node before scaling cluster-wide.
• - Treat storage readiness as a gating dependency for later burn-in tests.

Drill: Run a single-node storage sanity test and record minimum throughput thresholds for promotion.

Exam Decision Hierarchy

Prioritize decisions in this order: safety and hardware integrity, baseline consistency, controlled validation, then optimization.

• - If integrity checks fail, stop optimization and remediate first.
• - Compare against known-good baseline before changing multiple variables.
• - Document rationale for each decision to support incident replay.

Operational Evidence Standard

Treat every key action as evidence-producing: command, output, timestamp, and expected vs observed behavior.

• - Evidence should be reproducible by another engineer.
• - Use stable command templates for repeated environments.
• - Keep concise but complete validation artifacts for exam-style reasoning.

Bring-up gate design and evidence discipline

Treat bring-up as a gated workflow: each stage must produce artifacts before the next stage starts.

• - A gate should have a binary pass/fail rule and named owner.
• - Artifacts should include command outputs, firmware versions, and telemetry snapshots.
• - Gate failures should stop promotion to avoid compounding unresolved faults.

Topology-first thinking for AI factories

Topology choices determine collective communication behavior long before software tuning begins.

• - Match topology design to workload traffic shape and scaling profile.
• - Validate transceiver and cable classes against target bandwidth lanes.
• - Use topology review as a prerequisite for NCCL and ClusterKit test interpretation.

Firmware and platform readiness as one system

Firmware, thermals, power, and management-plane health must be validated as a single readiness package.

• - Mixed firmware state can mimic application-layer instability.
• - Thermal and power limits can invalidate benchmark confidence if unchecked.
• - BMC/OOB visibility should be operational before in-band troubleshooting.

Scenario A: New HGX node fails distributed validation on day 1

A newly installed node passes OS boot but fails first cluster communication tests. The goal is to isolate whether failure is physical, firmware, or runtime readiness.

        [BMC/OOB Network]
              |
[Mgmt Switch]---[Host Node]---[Top-of-Rack Switch]
                  |
               [HGX GPUs]
                  |
             [Storage Path]

nvidia-smi --query-gpu=name,temperature.gpu,power.draw,clocks.sm --format=csv

NVIDIA H100, 38 C, 185 W, 1410 MHz\nNVIDIA H100, 40 C, 191 W, 1410 MHz

mlxlink -d <device> --json

{\n  "state": "Active",\n  "speed": "400G",\n  "signal_quality": "Pass"\n}

Scenario B: Firmware update window introduces intermittent node instability

After a maintenance window, one node shows random communication failures. Objective is safe rollback-or-promote decision.

[Firmware Repo] -> [Staging Node] -> [Canary Node] -> [Production Node Group]
      |                |                |                 |
  version matrix    pre-checks      post-checks       burn-in gate

sudo nvfwupd --query --format table

Component        Version\nGPU FW           96.00.7A\nNVSwitch FW      28.42.1000\nBMC FW           24.05.01

Pre-benchmark readiness runbook

Use this sequence before any cluster-scale benchmark to eliminate common false negatives from bring-up gaps.

ipmitool -I lanplus -H <bmc_ip> -U <user> chassis status

System Power: on\nPower Overload: false\nCooling/Fan Fault: false

nvidia-smi --query-gpu=index,name,ecc.mode.current --format=csv

0, NVIDIA H100, Enabled\n1, NVIDIA H100, Enabled

nvidia-smi topo -m

GPU0  GPU1  CPU Affinity\nGPU0   X    NV2   0-55\nGPU1  NV2    X    0-55

• - Run with consistent ambient conditions and record timestamp.
• - Any non-pass result should block benchmark promotion.

Initial storage readiness runbook

Validate storage path behavior before interpreting training or verification performance.

mount | rg -i '<storage_mount>'

10.0.0.24:/ai-data on /mnt/ai-data type nfs4 (rw,hard,timeo=600)

fio --name=readcheck --directory=/mnt/ai-data --rw=read --bs=1M --size=8G --numjobs=4 --iodepth=16

READ: bw=11.2GiB/s, iops=11468, lat (msec): avg=5.34

• - Use the same dataset profile as planned benchmark workload.
• - Capture baseline to compare against later incident investigations.

Node passes boot but fails first NCCL communication test

Inconsistent performance across identical nodes

Post-upgrade instability after firmware change

Walkthrough: First-node bring-up to workload-ready state

Complete first-node onboarding with evidence-driven validation across management, hardware, and storage paths.

1. Validate BMC/OOB access and baseline health.

   ipmitool -I lanplus -H <bmc_ip> -U <user> sel list | tail -n 10

   Expected: No critical unresolved hardware events.

2. Confirm GPU inventory and topology visibility.

   nvidia-smi -L && nvidia-smi topo -m

   Expected: All expected GPUs visible with stable topology map.

3. Run thermal/power readiness check under light load.

   nvidia-smi --query-gpu=temperature.gpu,power.draw --format=csv

   Expected: Thermal and power values remain within policy thresholds.

4. Validate link/transceiver state for data-path correctness.

   mlxlink -d <device> --json

   Expected: All required links active at expected speed with pass signal quality.

5. Run storage sanity benchmark and record baseline.

   fio --name=sanity --directory=/mnt/ai-data --rw=read --bs=1M --size=4G --numjobs=2

   Expected: Throughput meets minimum readiness threshold for domain policy.

Lab A: End-to-end first node bring-up

Execute first-node bring-up with documented gate checks.

• - Run physical, firmware, and management checks in defined order.
• - Capture baseline telemetry before and after each major step.
• - Fail fast on gating issues and document corrective actions.

ipmitool -I lanplus -H <bmc_ip> -U <user> chassis status

System Power: on\nPower Overload: false\nCooling/Fan Fault: false

Lab B: Firmware baseline integrity

Build a consistent firmware/software baseline across components.

• - Inventory versions for host, GPU, switch, and DPU-adjacent components.
• - Execute one controlled upgrade simulation in staging.
• - Verify post-upgrade health and rollback readiness.

nvidia-smi --query-gpu=index,name,ecc.mode.current --format=csv

0, NVIDIA H100, Enabled\n1, NVIDIA H100, Enabled

Lab C: Topology and cable validation

Prove that physical connectivity matches intended design.

• - Validate transceiver and cable compatibility against design specs.
• - Run basic signal-quality and link-state checks.
• - Document topology mismatches before cluster-scale tests.

nvidia-smi topo -m

GPU0  GPU1  CPU Affinity\nGPU0   X    NV2   0-55\nGPU1  NV2    X    0-55

Lab D: Storage readiness sanity

Confirm storage path is operational before performance validation.

• - Validate mount and path-level accessibility.
• - Run quick throughput and latency spot checks.
• - Record safe baseline thresholds for cluster test entry.

mount | rg -i '<storage_mount>'

10.0.0.24:/ai-data on /mnt/ai-data type nfs4 (rw,hard,timeo=600)