Physical Layer Management

Track 1: BlueField platform role in AI infrastructure

The exam expects operational familiarity with DPU-backed networking and infrastructure control planes.

• - Understand where BlueField sits in data-path and management-path architecture.
• - Map BlueField functions to provisioning, security, and networking operations.
• - Verify BlueField firmware/software alignment with switch and host baselines.

Drill: Diagram one host-network path and annotate BlueField responsibilities in the flow.

Track 2: BlueField lifecycle operations

Domain scope includes configuration and management, not just conceptual understanding.

• - Establish a repeatable update and validation process for BlueField software and firmware.
• - Monitor basic health and connectivity after each lifecycle operation.
• - Treat BlueField changes as part of cluster-wide change control.

Drill: Write a BlueField maintenance runbook with pre-checks, execution controls, and verification steps.

Track 3: MIG fundamentals and partition strategy

MIG configuration is explicitly in scope and can affect utilization and isolation outcomes.

• - MIG partitions one GPU into isolated instances with dedicated compute and memory slices.
• - Profile choices should match workload shape and service-level expectations.
• - MIG planning differs across AI inference, AI training, and HPC usage patterns.

Drill: Given three workloads, assign MIG profiles and justify expected tradeoffs.

Track 4: MIG operations and observability

Correct MIG setup requires operational checks and stable inventory handling.

• - Use supported tooling to create, list, and validate MIG instances.
• - Track MIG allocation changes with auditability for multi-tenant environments.
• - Validate that scheduler/runtime stack maps jobs to intended MIG resources.

Drill: Perform a MIG create/validate/reset cycle and record all state transitions.

Track 5: AI vs HPC configuration tradeoffs

The blueprint highlights both AI and HPC contexts for physical-layer decisions.

• - Latency-sensitive AI inference may prioritize predictable partition behavior.
• - HPC or large-model workloads may require fewer or no partitions for full-GPU access.
• - Resource isolation policies should align with job classes and operational objectives.

Drill: Build an allocation policy table: workload type, MIG policy, and expected operational impact.

Track 6: Failure patterns in physical-layer management

Most operational regressions come from drift, undocumented changes, or profile mismatch.

• - Avoid undocumented ad-hoc profile changes across production nodes.
• - Reconcile BlueField and MIG state after maintenance windows.
• - Integrate physical-layer checks into release and incident workflows.

Drill: Create a post-maintenance audit checklist for BlueField and MIG state consistency.

Exam Decision Hierarchy

Prioritize decisions in this order: safety and hardware integrity, baseline consistency, controlled validation, then optimization.

• - If integrity checks fail, stop optimization and remediate first.
• - Compare against known-good baseline before changing multiple variables.
• - Document rationale for each decision to support incident replay.

Operational Evidence Standard

Treat every key action as evidence-producing: command, output, timestamp, and expected vs observed behavior.

• - Evidence should be reproducible by another engineer.
• - Use stable command templates for repeated environments.
• - Keep concise but complete validation artifacts for exam-style reasoning.

BlueField in the AI infrastructure stack

BlueField acts as a programmable data-path and infrastructure-control element that can influence networking, isolation, and observability outcomes.

• - Treat BlueField state as part of cluster baseline, not an isolated appliance.
• - Changes to BlueField stack should follow the same change control rigor as host firmware.
• - BlueField validation should include connectivity and lifecycle consistency checks.

MIG partitioning strategy for AI and HPC

MIG is a resource isolation mechanism, not just a utilization toggle. Profile choice must align with workload SLOs.

• - Inference workloads often benefit from predictable partition isolation.
• - Large training/HPC workloads may require full GPU or larger slices.
• - MIG policy should be encoded by workload class to avoid ad-hoc allocation drift.

State drift and governance

Most physical-layer incidents come from undocumented drift across node-level MIG or platform-level BlueField states.

• - Record before/after state for every maintenance operation.
• - Reconcile expected vs observed state at end of each change window.
• - Escalate discrepancies before higher-level benchmark validation.

Scenario A: Multi-tenant inference pool with MIG drift

A subset of nodes starts violating inference latency SLOs after maintenance. Investigation reveals inconsistent MIG profiles across nodes.

[Scheduler] -> [Node Group A: MIG 1g/2g mix] -> [Inference Services]
               [Node Group B: drifted profiles] -> [Latency spikes]

nvidia-smi mig -lgi

GPU 0: GI 1 Profile 1g.10gb\nGPU 0: GI 2 Profile 2g.20gb

Scenario B: BlueField maintenance causes partial network inconsistency

After BlueField update, a node subset exhibits unstable communication despite healthy host-level checks.

[Control Plane]
   |
[BlueField Set A: baseline] --- [Switch Fabric] --- [BlueField Set B: updated]
                   \            intermittent path           /
                             [GPU Hosts]

ethtool <iface> | rg -i 'Speed|Link detected'

Speed: 400000Mb/s\nLink detected: yes

MIG lifecycle verification runbook

Use this runbook when enabling, validating, or resetting MIG layout for workload classes.

sudo nvidia-smi -i 0 -mig 1

Enabled MIG mode for GPU 00000000:17:00.0

nvidia-smi mig -lgip

GPU instance profiles:\n1g.10gb\n2g.20gb\n3g.40gb

nvidia-smi mig -lgi

GPU 0: GI 1 Profile 1g.10gb\nGPU 0: GI 2 Profile 2g.20gb

• - Reboot or service restart behavior depends on platform policy; validate after change.
• - Always reconcile instance inventory with scheduler/resource manager.

Physical-link readiness runbook

Validate link-level health when physical-layer regressions are suspected.

ethtool <iface>

Speed: 400000Mb/s\nDuplex: Full\nLink detected: yes

mlxlink -d <device> --json

{\n  "state": "Active",\n  "BER": "Within threshold",\n  "signal_quality": "Pass"\n}

• - Run both baseline and suspect nodes for side-by-side comparison.
• - Escalate any signal-quality anomaly before workload-level debugging.

MIG profile drift across node pool

BlueField subset inconsistency after lifecycle update

Workload class policy not aligned with MIG design

Walkthrough: MIG policy implementation for mixed AI workloads

Implement and validate a workload-class MIG policy with reproducible inventory state.

1. Enable MIG mode for target GPUs.

   sudo nvidia-smi -i 0 -mig 1

   Expected: MIG mode enabled successfully.

2. Create planned instance profiles for each class.

   nvidia-smi mig -cgi <profile_id> -C

   Expected: GPU instances created matching plan.

3. Validate created inventory against design.

   nvidia-smi mig -lgi

   Expected: Inventory output equals policy baseline.

4. Run one validation workload per class.

   python run_validation.py --class <class_name>

   Expected: Each class runs within expected performance bounds.

5. Record baseline and integrate with scheduler metadata.

   Expected: Resource definitions and node states are synchronized.

Lab A: BlueField baseline validation

Confirm BlueField platform readiness in a controlled environment.

• - Validate current software and firmware baseline.
• - Run health and connectivity checks after baseline capture.
• - Document a maintenance-safe rollback point.

sudo nvidia-smi -i 0 -mig 1

Enabled MIG mode for GPU 00000000:17:00.0

Lab B: MIG profile operations

Practice deterministic MIG lifecycle management.

• - Enable MIG mode and create selected instance profiles.
• - Validate visible inventory and mapping behavior.
• - Reset to baseline state and confirm cleanup.

nvidia-smi mig -lgip

GPU instance profiles:\n1g.10gb\n2g.20gb\n3g.40gb

Lab C: Policy-by-workload mapping

Translate workload requirements into physical-layer configuration policy.

• - Classify workloads by latency, throughput, and isolation needs.
• - Select MIG or full-GPU policy per class.
• - Capture expected operational tradeoffs.

nvidia-smi mig -lgi

GPU 0: GI 1 Profile 1g.10gb\nGPU 0: GI 2 Profile 2g.20gb

Lab D: Drift detection and recovery

Detect and remediate state drift after simulated maintenance.

• - Compare expected vs observed BlueField/MIG states.
• - Apply controlled remediation steps.
• - Record evidence and update runbook controls.

ethtool <iface>

Speed: 400000Mb/s\nDuplex: Full\nLink detected: yes