NVIDIA InfiniBand Networking

Track 1: InfiniBand architecture essentials

You need to reason about InfiniBand behavior and management surfaces before tuning commands.

• - Understand subnet management and control-plane role in fabric health.
• - Map collective communication sensitivity to path and congestion behavior.
• - Separate host, switch, and management responsibilities in diagnostics.

Drill: Explain which InfiniBand components you would check first for cluster-wide communication regressions.

Track 2: UFM-based configuration and validation

Blueprint scope explicitly requires configuring and validating InfiniBand using UFM.

• - Use UFM for inventory, topology state, and policy-aware operations.
• - Validate post-change topology and link-state consistency.
• - Preserve rollback-capable evidence before large-scale updates.

Drill: Run a UFM health snapshot and identify one high-priority warning class.

Track 3: CLI monitoring and optimization

Command-line diagnostics remain critical for rapid fault isolation and exam-style troubleshooting.

• - Use InfiniBand CLI tools to verify link state, errors, and throughput behavior.
• - Correlate host-level and switch-level indicators when diagnosing bottlenecks.
• - Tune only after confirming topology and firmware baseline integrity.

Drill: Create a command sequence to isolate whether issue is endpoint, path, or congestion related.

Track 4: Security and multi-tenancy

Partitioning and access controls are required for multi-team AI clusters with shared infrastructure.

• - Apply partitioning strategy for tenant separation.
• - Validate management and operations access boundaries.
• - Audit policy consistency after fabric changes.

Drill: Draft a partition and access model for two tenant groups sharing one fabric.

Track 5: Congestion and bottleneck troubleshooting

A dedicated objective covers congestion and bottleneck optimization.

• - Differentiate transient spikes from sustained congestion.
• - Use targeted tests to locate hot links or misrouted traffic.
• - Confirm improvement with repeat validation, not one-time samples.

Drill: Design a congestion triage loop that ends with measurable acceptance criteria.

Exam Decision Hierarchy

Prioritize decisions in this order: safety and hardware integrity, baseline consistency, controlled validation, then optimization.

• - If integrity checks fail, stop optimization and remediate first.
• - Compare against known-good baseline before changing multiple variables.
• - Document rationale for each decision to support incident replay.

Operational Evidence Standard

Treat every key action as evidence-producing: command, output, timestamp, and expected vs observed behavior.

• - Evidence should be reproducible by another engineer.
• - Use stable command templates for repeated environments.
• - Keep concise but complete validation artifacts for exam-style reasoning.

Evidence stack for InfiniBand troubleshooting

Reliable diagnosis combines management-plane visibility with low-level path and endpoint diagnostics.

• - Start from topology state, then move to path-level counters.
• - Anchor diagnostics to workload time windows.
• - Record all remediation decisions with expected impact.

Congestion as a systems problem

Congestion often emerges from interacting workload, routing, and policy conditions rather than one broken link.

• - Measure persistence and locality of pressure before tuning.
• - Validate that tuning does not break isolation or reliability.
• - Retest under representative load after each change.

Partitioning with operations in mind

Security controls should enforce tenant boundaries while preserving controlled operations workflows.

• - Keep management access paths explicit and auditable.
• - Validate partition membership continuously, not only at creation.
• - Treat policy drift as ongoing operational risk.

Scenario: Fabric-level latency spikes under nightly training load

Nightly distributed training jobs exhibit intermittent latency spikes and reduced all-reduce efficiency.

GPU Pods
   |
InfiniBand Fabric
   |-- Core Switches
   |-- Edge Switches
   |-- Management via UFM

ibstat

All relevant ports report Active state and expected link rate.

ibdiagnet -v

No critical path errors; warnings are mapped to specific remediation candidates.

Scenario: New tenant cannot communicate with assigned resources

A newly provisioned tenant reports communication failures across assigned compute resources.

Tenant A Partition
Tenant B Partition
   |
Shared InfiniBand Fabric
   |
UFM + Access Control

InfiniBand health baseline runbook

Capture baseline link and topology health before advanced troubleshooting.

ibstat

Ports are Active with expected physical and link layer attributes.

ibnetdiscover | head -n 40

Topology inventory aligns with expected node and switch map.

ufm_health --summary

Management summary highlights stable fabric state or specific warning categories.

• - Run before and after any fabric policy or routing changes.
• - Store snapshots with timestamps tied to workload events.

Congestion and bottleneck triage runbook

Localize and remediate sustained bandwidth or latency bottlenecks.

ib_write_bw -d mlx5_0 -F --report_gbits <peer_host>

Measured bandwidth indicates whether path performance meets baseline.

perfquery -x

Counter output reveals error trends and congestion-linked symptoms.

saquery -s

Service and path queries return expected records for active fabric routes.

• - Re-run after each corrective action to validate improvement.
• - Avoid broad tuning without locating persistent hot links first.

Intermittent all-reduce slowdown during peak windows

Tenant partition misconfiguration after onboarding

Fabric appears healthy but workload still underperforms

Walkthrough: UFM-led InfiniBand validation cycle

Use UFM and CLI diagnostics to validate fabric readiness and isolate candidate bottlenecks.

1. Capture management-plane health snapshot.

   ufm_health --summary

   Expected: Health summary identifies stable fabric or actionable warnings.

2. Validate endpoint link state.

   ibstat

   Expected: Relevant ports are Active and configured as expected.

3. Run topology/path diagnostics.

   ibdiagnet -v

   Expected: Critical path errors are absent or isolated to explicit links.

4. Measure path bandwidth baseline.

   ib_write_bw -d mlx5_0 -F --report_gbits <peer_host>

   Expected: Bandwidth remains inside accepted baseline range.

Walkthrough: Partition and isolation verification

Validate security and multi-tenant behavior without degrading operations access.

1. Query partition and service state.

   saquery -s

   Expected: Service entries and partition records are present as expected.

2. Run intra-tenant connectivity checks.

   ibping -S && ibping -c <target_guid>

   Expected: Allowed tenant paths succeed with stable response.

3. Run cross-tenant validation check.

   ibping -c <other_tenant_guid>

   Expected: Disallowed path is blocked per policy design.

Lab A: UFM-driven fabric health baseline

Capture fabric topology and health state in UFM and validate consistency with expected design.

• - Inventory nodes, links, and switch states.
• - Record warnings and classify by severity.
• - Confirm topology consistency with documented architecture.

ibstat

Ports are Active with expected physical and link layer attributes.

Lab B: CLI-based throughput and error triage

Use InfiniBand commands to isolate performance regression source.

• - Run link and error counters on suspect paths.
• - Run bandwidth probe across candidate endpoints.
• - Correlate results with workload timing window.

ibnetdiscover | head -n 40

Topology inventory aligns with expected node and switch map.

Lab C: Partition and tenant policy validation

Verify multi-tenant controls are applied and enforceable.

• - Apply partition policy and verify endpoint membership.
• - Test allowed and denied communication paths.
• - Review management access boundaries.

ufm_health --summary

Management summary highlights stable fabric state or specific warning categories.

Lab D: Congestion hotspot investigation

Locate and remediate congestion hotspots with evidence-driven tuning.

• - Identify sustained high-pressure links.
• - Apply targeted adjustment and rerun validation tests.
• - Confirm stability across multiple windows.

ib_write_bw -d mlx5_0 -F --report_gbits <peer_host>

Measured bandwidth indicates whether path performance meets baseline.