AI Data Center Design and Optimization

Track 1: AI data center architecture fundamentals

Exam scope expects architecture-first reasoning before implementation and tuning decisions.

• - Differentiate compute, network, storage, and management responsibilities in data center design.
• - Map east-west collective traffic versus north-south service traffic.
• - Use a deterministic deployment and validation sequence before workload onboarding.

Drill: Given one training and one inference workload, describe dominant paths and first validation checks.

Track 2: Topology design by use case

Topology mistakes cause throughput ceilings and debugging complexity later in operations.

• - Use leaf-spine as baseline and scale rail-aware fabric design for large GPU clusters.
• - Evaluate oversubscription impact on all-reduce heavy distributed training.
• - Include failure-domain boundaries in topology planning.

Drill: Create a two-tier topology sketch and mark where congestion risk appears first.

Track 3: Storage network architecture

Storage path design directly affects data loader throughput and end-to-end iteration time.

• - Separate storage traffic class from collective communication when possible.
• - Choose storage protocol and network path based on required throughput and latency profile.
• - Validate that storage network design matches checkpoint and dataset access behavior.

Drill: For a 70B model training job, list storage-path checks needed before scale-out run.

Track 4: Power and cooling validation discipline

Power/cooling issues can invalidate performance and stability assumptions before networking is fully stressed.

• - Validate inlet temperature, airflow direction, and rack power headroom before scale tests.
• - Correlate thermal or power throttling events with workload throughput behavior.
• - Use CLI and telemetry checks as release gates prior to production cutover.

Drill: Define a go/no-go checklist for power and cooling before 32+ GPU scale tests.

Track 5: Architecture validation criteria

You need measurable criteria to confirm architecture decisions before production deployment.

• - Use synthetic communication tests plus representative workload traces.
• - Define acceptance thresholds for latency, bandwidth, and error budgets.
• - Document go/no-go criteria per topology domain.

Drill: Write a three-metric acceptance gate for promoting architecture design to implementation.

Exam Decision Hierarchy

Prioritize decisions in this order: safety and hardware integrity, baseline consistency, controlled validation, then optimization.

• - If integrity checks fail, stop optimization and remediate first.
• - Compare against known-good baseline before changing multiple variables.
• - Document rationale for each decision to support incident replay.

Operational Evidence Standard

Treat every key action as evidence-producing: command, output, timestamp, and expected vs observed behavior.

• - Evidence should be reproducible by another engineer.
• - Use stable command templates for repeated environments.
• - Keep concise but complete validation artifacts for exam-style reasoning.

Architecture-first troubleshooting prevention

Most network incidents in AI clusters are architecture debt surfacing during scale or tenant growth.

• - Capture assumptions on traffic ratios before deployment.
• - Define acceptable oversubscription boundaries explicitly.
• - Treat storage and collective paths as co-equal design constraints.

Topology decisions as business risk controls

Topology defines not only performance but also operational risk and recovery behavior.

• - Failure-domain boundaries should map to operational ownership.
• - Spare capacity planning matters for remediation events.
• - Recovery speed should be evaluated during design, not after outage.

Storage-network coupling in AI pipelines

Storage throughput and latency variance directly affects GPU utilization and job completion time.

• - Checkpoint and dataset access patterns are often different and need separate validation.
• - Test with realistic concurrency, not single-stream benchmarks only.
• - Include storage-path metrics in architecture acceptance gates.

Scenario: Training cluster with unstable scaling performance

A 64-GPU training cluster performs well at 8 GPUs but degrades sharply at 32+ GPUs. You need to determine whether the issue is topology, storage, or scheduling-driven.

Clients
  |
API/Gateway
  |
Leaf-Spine Fabric
  |-- GPU Train Nodes
  |-- Storage Nodes
  |-- Management/Observability

nv show interface && lldpcli show neighbors

All planned links up, expected neighbor map present, no unexpected peer changes.

fio --name=readcheck --directory=/mnt/dataset --rw=read --bs=1M --size=8G --numjobs=4

Stable throughput with low variance across test windows.

Scenario: New tenant onboarding introduces intermittent packet loss

A new tenant environment was added and existing tenant workloads show intermittent failures during peak windows.

Tenant A VRF ---|
Tenant B VRF ---|--- Shared Spine
Mgmt VRF -------|
Storage Fabric --|

Topology baseline runbook

Capture architecture-grounded baseline before any optimization or remediation decision.

nv show interface

Interfaces in expected admin/oper state with planned speed and MTU values.

lldpcli show neighbors

Neighbor inventory aligns with documented topology.

ip route show | head -n 20

Expected route entries present for tenant and management paths.

• - Capture snapshots before and after architecture changes.
• - Use the same command set across all fabric zones for comparability.

Storage-path validation runbook

Validate storage network assumptions used by model training and checkpoint workflows.

ping -c 20 <storage_endpoint>

Latency stays within expected band with low packet loss.

iperf3 -c <storage_or_gateway_host> -P 4 -t 20

Aggregate throughput remains stable across parallel streams.

fio --name=dataset --directory=/mnt/dataset --rw=randread --bs=256k --size=4G --numjobs=8

IO profile meets minimum throughput and latency targets.

• - Run during both idle and peak windows to detect contention.
• - Pair network metrics with loader step-time telemetry when possible.

Good single-node results but poor multi-node scaling

Checkpoint spikes causing periodic training stalls

Tenant onboarding introduces cross-tenant instability

Walkthrough: Architecture validation for new training environment

Confirm topology and storage-network choices meet scale-out training requirements before production activation.

1. Collect baseline link and neighbor state.

   nv show interface && lldpcli show neighbors

   Expected: Link inventory matches design with no missing adjacencies.

2. Measure storage-path network behavior.

   iperf3 -c <storage_host> -P 4 -t 20

   Expected: Throughput is stable and aligns with design target.

3. Run dataset read profile to emulate training ingest.

   fio --name=ingest --directory=/mnt/dataset --rw=read --bs=1M --size=8G --numjobs=4

   Expected: Read profile shows acceptable throughput and variance.

4. Execute scaled test and compare against baseline.

   python3 run_scale_probe.py --nodes 8,16,32

   Expected: Scaling behavior follows planned threshold envelope.

Walkthrough: Multi-tenant segmentation verification

Verify tenant isolation model and operations visibility path for day-2 support.

1. Validate route and segmentation boundaries.

   ip route show | grep -E 'tenant-a|tenant-b'

   Expected: Tenant routes map to intended isolated domains.

2. Run controlled cross-tenant connectivity check.

   ping -c 5 <tenant_b_endpoint_from_tenant_a>

   Expected: Disallowed flow is blocked by policy.

3. Validate approved management observability flow.

   curl -I http://<metrics-endpoint>/health

   Expected: Monitoring path succeeds without violating tenant boundaries.

Lab A: Build workload traffic matrix

Translate a workload description into network traffic classes and priority paths.

• - Classify east-west, north-south, and storage traffic.
• - Map each traffic class to target interfaces and bandwidth profile.
• - Identify first-hop bottleneck risk under scale-out.

nv show interface

Interfaces in expected admin/oper state with planned speed and MTU values.

Lab B: Topology tradeoff drill

Compare two topology options and choose one with explicit decision criteria.

• - Define oversubscription and failure domain assumptions.
• - Estimate impact on collective communication latency.
• - Recommend topology with supporting rationale.

lldpcli show neighbors

Neighbor inventory aligns with documented topology.

Lab C: Storage path architecture validation

Validate storage-network suitability for checkpoint-heavy training pipeline.

• - Measure storage read path throughput baseline.
• - Observe network contention during synthetic IO bursts.
• - Document tuning levers for storage-network consistency.

ip route show | head -n 20

Expected route entries present for tenant and management paths.

Lab D: Multi-tenant segmentation plan

Design data center pre-flight validation for power/cooling and readiness gates.

• - Define power and cooling telemetry checkpoints before workload launch.
• - Map rack-level constraints to rollout and scheduling guardrails.
• - Produce a go/no-go sheet for pre-production validation.

ping -c 20 <storage_endpoint>

Latency stays within expected band with low packet loss.