Installation and Deployment

Track 1: Deployment sequence and prerequisites

This is the highest-weight domain and depends on strict sequencing from infrastructure readiness to workload runtime.

• - Validate hardware and software prerequisites before platform bootstrap.
• - Use an explicit deployment sequence with gate checks at each layer.
• - Capture baseline artifacts for post-deployment validation.

Drill: Write a deployment runbook with stop conditions for each layer (firmware, OS, scheduler, runtime).

Track 2: Management and monitoring stack installation

Blueprint scope explicitly includes BCM, Mission Control, and UFM integration.

• - Differentiate management stack roles and integration touchpoints.
• - Validate health across management endpoints after installation.
• - Plan rollback for failed management stack updates.

Drill: Document one end-to-end stack validation flow from install to health check.

Track 3: Scheduler and runtime setup

Run:ai, Slurm, and Kubernetes scheduler setup determines workload routing quality.

• - Install and validate scheduler stack in deterministic order.
• - Ensure NVIDIA container runtime/toolkit is configured on worker nodes.
• - Verify GPU visibility from orchestrated workloads.

Drill: Deploy one test workload and verify scheduler placement plus GPU runtime readiness.

Track 4: Registry, model runtime, and inference endpoint

NGC registry/API key, NIM, and TensorRT-LLM setup are explicit objectives and critical for production serving.

• - Validate registry authentication and model artifact access.
• - Install and verify NIM/TensorRT-LLM runtime dependencies.
• - Expose and validate inference endpoints with health checks.

Drill: Configure one model-serving endpoint and prove readiness with a structured test call.

Track 5: DOCA and Magnum IO service integration

Lower-layer runtime services affect network and workload performance characteristics.

• - Install container toolkit and runtime dependencies on worker nodes.
• - Install and validate DOCA services on DPU Arm where required.
• - Confirm Magnum IO stack readiness for target workload path.

Drill: Build a dependency map showing how DOCA/container toolkit/Magnum IO affect workload startup.

Exam Decision Hierarchy

Prioritize decisions in this order: safety and hardware integrity, baseline consistency, controlled validation, then optimization.

• - If integrity checks fail, stop optimization and remediate first.
• - Compare against known-good baseline before changing multiple variables.
• - Document rationale for each decision to support incident replay.

Operational Evidence Standard

Treat every key action as evidence-producing: command, output, timestamp, and expected vs observed behavior.

• - Evidence should be reproducible by another engineer.
• - Use stable command templates for repeated environments.
• - Keep concise but complete validation artifacts for exam-style reasoning.

Deployment as dependency graph

Treat installation as a dependency graph, not a linear script, so failure handling stays predictable.

• - Every node in the graph should have validation criteria.
• - Upstream failures should block downstream execution.
• - Dependency graph should include external services like registry access.

Control-plane versus runtime readiness

Control-plane availability does not guarantee workload runtime correctness.

• - Validate scheduler placement and runtime execution separately.
• - Confirm GPU visibility from workload context, not host only.
• - Capture runtime logs as baseline artifacts.

Endpoint readiness discipline

Model-serving readiness must include artifact access, model load, endpoint health, and response validation.

• - Smoke tests should include at least one real inference payload.
• - Health checks alone are insufficient without response validation.
• - Keep a known-good sample request for regression checks.

Scenario: Fresh deployment with partial runtime failure

Control plane services are up, but GPU workloads fail on a subset of worker nodes after deployment.

Mgmt Stack (BCM/Mission Control/UFM)
            |
Scheduler Layer (Run:ai/Slurm/K8s)
            |
Worker Nodes + GPU Runtime

kubectl get nodes -o wide && kubectl describe node <worker-node>

Node is Ready with expected GPU resource advertisement.

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Container sees expected GPU devices and driver/runtime versions.

Scenario: Endpoint deploy succeeds but inference fails

NIM/TensorRT-LLM service is running, yet inference requests fail intermittently.

NGC Registry/Auth
      |
Model Runtime (NIM/TensorRT-LLM)
      |
Inference Endpoint/API

Deployment gate verification runbook

Validate each installation stage before promoting to next stage.

cmsh -c 'show version'

BCM CLI responds with installed version details.

kubectl get nodes -o wide

All required nodes are Ready with expected roles.

nvidia-ctk runtime configure --runtime=docker

Runtime configuration updated successfully with no errors.

• - Run and archive outputs per stage.
• - Stop progression if one stage fails validation.

Registry and endpoint readiness runbook

Validate artifact access and serving path before production onboarding.

ngc config current

Active org/team and API key context are valid.

curl -sS http://<nim-endpoint>/health

Health payload indicates ready state.

curl -sS -X POST http://<nim-endpoint>/v1/completions -H 'Content-Type: application/json' -d '{"prompt":"hello","max_tokens":8}'

Endpoint returns valid completion payload with expected schema.

• - Pair health checks with inference tests.
• - Capture latency and error rate baseline for post-deploy comparison.

Scheduler appears healthy but workloads remain Pending

Model-serving endpoint unhealthy after deploy

DPU-side services installed but workload path unstable

Walkthrough: Installation-to-workload validation

Run a full deployment path from control-plane install to successful GPU workload execution.

1. Validate cluster node readiness.

   kubectl get nodes -o wide

   Expected: All required nodes are Ready.

2. Confirm GPU runtime in container context.

   docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

   Expected: Container reports expected GPU inventory.

3. Submit validation workload.

   kubectl apply -f gpu-smoke-test.yaml && kubectl logs -f job/gpu-smoke-test

   Expected: Workload completes successfully with no GPU runtime errors.

Walkthrough: Registry to inference endpoint

Validate NGC access, model runtime startup, and endpoint response quality.

1. Validate NGC config context.

   ngc config current

   Expected: Active config shows valid org/team and API setup.

2. Check endpoint health.

   curl -sS http://<nim-endpoint>/health

   Expected: Health output indicates ready.

3. Run inference smoke test.

   curl -sS -X POST http://<nim-endpoint>/v1/completions -H 'Content-Type: application/json' -d '{"prompt":"test","max_tokens":8}'

   Expected: Endpoint returns valid completion response.

Lab A: End-to-end deployment gate rehearsal

Execute full installation sequence with explicit gate evidence.

• - Run prerequisite checks before any package installation.
• - Install stack in documented sequence with verification per stage.
• - Record rollback points and outcome of each gate.

cmsh -c 'show version'

BCM CLI responds with installed version details.

Lab B: Scheduler and GPU runtime validation

Confirm scheduler stack and worker runtime can launch GPU workloads.

• - Validate scheduler control-plane health.
• - Launch test GPU workload and verify placement.
• - Collect runtime logs for post-deployment baseline.

kubectl get nodes -o wide

All required nodes are Ready with expected roles.

Lab C: Registry to model-serving path

Validate private registry access and inference endpoint readiness.

• - Authenticate to NGC and pull required artifact.
• - Deploy NIM/TensorRT-LLM service path.
• - Run health and inference smoke tests.

nvidia-ctk runtime configure --runtime=docker

Runtime configuration updated successfully with no errors.

Lab D: DOCA and Magnum IO integration check

Verify lower-layer service dependencies for workload communication path.

• - Confirm DOCA service state on DPU Arm node.
• - Validate worker container runtime alignment.
• - Run representative communication test and record outcome.

ngc config current

Active org/team and API key context are valid.