Kubernetes Integration

Track 1: Kubernetes architecture for AI networking

You need to understand where orchestration interacts with networking and GPU scheduling before troubleshooting symptoms.

• - Control-plane, worker-plane, and CNI responsibilities are separate failure domains.
• - Pod scheduling, device plugins, and CNI pathing jointly determine workload readiness.
• - Node labels, taints, and runtime classes directly affect GPU workload placement.

Drill: Map one training workload from API submit to pod networking and GPU allocation steps.

Track 2: Cluster/node prerequisites and baseline validation

Many Kubernetes networking failures originate in node prerequisites, not in overlay or routing logic.

• - Kernel modules, container runtime, and kubelet settings must match cluster standard.
• - Time sync, DNS reachability, and MTU consistency are required before CNI validation.
• - GPU nodes require deterministic driver/runtime compatibility for stable operator behavior.

Drill: Create a pre-flight checklist and run it on one control-plane and one worker node.

Track 3: NVIDIA GPU Operator and node readiness

The exam explicitly expects operator usage and operational verification patterns.

• - Validate operator-managed components: driver, toolkit, device plugin, and DCGM exporter.
• - Distinguish operator rollout failure from CNI or scheduler issues.
• - Confirm allocatable GPU resources at node and namespace scope.

Drill: Run end-to-end validation proving a GPU pod is admitted and can see expected devices.

Track 4: CNI plugin and Kubernetes network integration

Network plugin misconfiguration can mimic compute or application failures.

• - Know CNI config locations and plugin chain order on worker nodes.
• - Validate network policy behavior separately from CNI data-path health.
• - Understand secondary interface patterns for high-performance data paths.

Drill: Test pod-to-pod, pod-to-service, and node-to-pod connectivity with one policy-enabled namespace.

Track 5: kubectl diagnostics and incident workflow

Fast, deterministic command flow is required for exam scenarios and production incidents.

• - Use describe/events/logs to isolate scheduling vs runtime vs network causes.
• - Capture node condition, daemonset health, and namespace-scoped policy state.
• - Tie every remediation to objective before/after validation evidence.

Drill: Resolve a pending GPU pod incident and produce a short root-cause timeline.

Exam Decision Hierarchy

Prioritize decisions in this order: safety and hardware integrity, baseline consistency, controlled validation, then optimization.

• - If integrity checks fail, stop optimization and remediate first.
• - Compare against known-good baseline before changing multiple variables.
• - Document rationale for each decision to support incident replay.

Operational Evidence Standard

Treat every key action as evidence-producing: command, output, timestamp, and expected vs observed behavior.

• - Evidence should be reproducible by another engineer.
• - Use stable command templates for repeated environments.
• - Keep concise but complete validation artifacts for exam-style reasoning.

Orchestrator and network coupling

Kubernetes integration failures often appear as application incidents but originate from node, CNI, or policy coupling issues.

• - Always classify failures by layer before applying fixes.
• - Node readiness and CNI readiness are separate checks.
• - Operator health does not guarantee workload path health.

Deterministic kubectl triage

Consistent command flow shortens mean-time-to-isolation and improves exam response quality.

• - Start with pod status and events.
• - Validate node conditions and allocatable resources next.
• - Confirm policy and service routing only after baseline checks.

Policy-safe performance operations

Performance fixes must preserve namespace isolation and intended policy behavior.

• - Do not bypass policy controls to mask data-path defects.
• - Validate allowed and denied flows after every change.
• - Keep rollback conditions explicit before policy updates.

Scenario: GPU training pods stay Pending after cluster update

A cluster update completed, but new training pods requesting GPUs remain Pending in one namespace.

API Server
  |
Scheduler ---- etcd
  |
GPU Worker Nodes
  |
CNI Plugin + GPU Operator Components

kubectl describe pod <pod> -n <ns> && kubectl get events -n <ns> --sort-by=.lastTimestamp

Scheduling or resource errors are explicit and time-correlated.

kubectl get nodes -o custom-columns=NAME:.metadata.name,ALLOC:.status.allocatable.nvidia\.com/gpu

Node allocatable GPU values match expected capacity.

Scenario: Pod-to-pod traffic fails across namespaces after policy change

A namespace policy hardening rollout was applied; cross-namespace service calls now fail intermittently.

Namespace A Pods ---- Service A
Namespace B Pods ---- Service B
        |
CNI Plugin + NetworkPolicy Engine

Runbook: Cluster and GPU readiness baseline

Establish whether issue source is scheduler/resource/operator before CNI tuning.

kubectl get nodes -o wide && kubectl get pods -n kube-system

All required nodes are Ready and system pods are healthy.

kubectl get pods -n gpu-operator && kubectl logs -n gpu-operator -l app=nvidia-device-plugin-daemonset --tail=80

Operator components are Running with no device registration errors.

• - If nodes are NotReady, resolve node/runtime baseline before CNI troubleshooting.
• - If operator is unhealthy, remediation should focus on GPU stack lifecycle first.

Runbook: CNI and policy troubleshooting

Isolate data-path versus policy enforcement failures.

kubectl get networkpolicy -A

Policy set matches intended namespace segmentation model.

kubectl exec -n <ns> <pod> -- sh -c 'nc -vz <target> <port>'

Connectivity result aligns with expected allow/deny behavior.

• - Use both positive and negative tests for policy validation.
• - Capture timestamped evidence before and after changes.

GPU Operator components remain CrashLoopBackOff

Pod networking unstable after CNI update

Walkthrough: Validate GPU Operator readiness

Confirm end-to-end GPU workload admission and device visibility in Kubernetes.

1. Verify cluster and operator baseline health.

   kubectl get nodes && kubectl get pods -n gpu-operator

   Expected: Nodes are Ready and operator components are Running.

2. Launch a GPU validation pod.

   kubectl apply -f gpu-smoke-test.yaml

   Expected: Pod schedules onto GPU node and enters Running state.

3. Confirm GPU visibility from inside pod.

   kubectl exec -it <gpu-pod> -- nvidia-smi

   Expected: Expected GPU inventory is visible without runtime errors.

Walkthrough: CNI and network policy validation

Prove intended connectivity behavior across namespaces with policy controls.

1. List active policies and namespace bindings.

   kubectl get networkpolicy -A

   Expected: Policies appear in intended namespaces with expected selectors.

2. Run allow-path connectivity test.

   kubectl exec -n ns-a pod-a -- nc -vz service-b.ns-b.svc.cluster.local 8080

   Expected: Allowed path succeeds.

3. Run deny-path connectivity test.

   kubectl exec -n ns-a pod-a -- nc -vz denied-target.ns-b.svc.cluster.local 8080

   Expected: Denied path fails as intended.

Lab A: Baseline cluster health and node prerequisites

Prove control-plane and worker nodes are ready before CNI/operator work.

• - Validate node readiness and kube-system pod health.
• - Confirm runtime, DNS, and clock state are consistent across nodes.
• - Record baseline outputs for comparison during incident drills.

kubectl get nodes -o wide && kubectl get pods -n kube-system

All required nodes are Ready and system pods are healthy.

Lab B: GPU Operator validation lab

Deploy and validate operator-managed components on GPU workers.

• - Verify operator deployment and daemonset status.
• - Launch a GPU test pod and confirm device visibility.
• - Collect logs/events for failure-path handling.

kubectl get pods -n gpu-operator && kubectl logs -n gpu-operator -l app=nvidia-device-plugin-daemonset --tail=80

Operator components are Running with no device registration errors.

Lab C: CNI and policy behavior lab

Validate data-path and policy behavior with deterministic checks.

• - Inspect CNI configuration and plugin chain on worker nodes.
• - Run cross-namespace connectivity tests with policy toggles.
• - Confirm expected allow/deny behavior with packet evidence.

kubectl get networkpolicy -A

Policy set matches intended namespace segmentation model.

Lab D: kubectl-led incident response lab

Diagnose and remediate a simulated cluster networking incident.

• - Identify symptom class from events and pod status.
• - Narrow root cause to scheduler, runtime, CNI, or policy layer.
• - Apply one fix and validate workload recovery.

kubectl exec -n <ns> <pod> -- sh -c 'nc -vz <target> <port>'

Connectivity result aligns with expected allow/deny behavior.