Automation and Configuration

Track 1: Configuration-at-scale fundamentals

Manual configuration does not scale for AI fabrics; deterministic automation is required.

• - Configuration intent must be template-driven and version-controlled.
• - Pre-check, change, and post-check phases should be explicit in every rollout.
• - Rollback criteria must be defined before execution.

Drill: Design a three-phase change workflow for a 32-switch maintenance window.

Track 2: Automation tools and control surfaces

The blueprint expects practical use of tools to automate and scale configuration tasks.

• - Use automation frameworks to enforce consistent config deployment and validation.
• - Separate source-of-truth, rendered config, and runtime state checks.
• - Use idempotent operations to avoid unintended repeated changes.

Drill: Create an automation inventory and map each tool to its control responsibility.

Track 3: CLI validation and optimization loops

Automation does not remove CLI responsibility; CLI still validates runtime reality.

• - After automation rollout, verify interfaces, routes, and policy state with CLI checks.
• - Tie config changes to measurable network performance outcomes.
• - Use staged rollout to reduce blast radius.

Drill: Run one staged change in lab and produce before/after validation evidence.

Track 4: Drift detection and policy compliance

Configuration drift is a top source of recurring incidents in multi-team environments.

• - Regularly compare intended config with live device state.
• - Classify drift by risk level and remediation urgency.
• - Enforce policy checks before change promotion.

Drill: Build a drift report format that includes severity, owner, and remediation SLA.

Track 5: Safe change management for AI workloads

AI cluster traffic is sensitive to network instability; change safety directly protects workload uptime.

• - Use canary scope and maintenance windows for high-risk modifications.
• - Define stop/rollback conditions linked to objective metrics.
• - Capture change evidence for audit and incident replay.

Drill: Define go/no-go and rollback gates for a fabric-wide configuration update.

Exam Decision Hierarchy

Prioritize decisions in this order: safety and hardware integrity, baseline consistency, controlled validation, then optimization.

• - If integrity checks fail, stop optimization and remediate first.
• - Compare against known-good baseline before changing multiple variables.
• - Document rationale for each decision to support incident replay.

Operational Evidence Standard

Treat every key action as evidence-producing: command, output, timestamp, and expected vs observed behavior.

• - Evidence should be reproducible by another engineer.
• - Use stable command templates for repeated environments.
• - Keep concise but complete validation artifacts for exam-style reasoning.

Intent vs runtime state

Template/render output defines intent, but operational correctness is determined by live runtime state.

• - Validate intent and runtime separately.
• - Use post-checks that directly query device state.
• - Treat divergence as operational risk, not cosmetic drift.

Change safety economics

Canary rollout and rollback planning reduce outage blast radius and recovery cost.

• - High-risk changes should start with minimal scope.
• - Promotion requires objective gate success.
• - Rollback should be executable in minutes, not hours.

Automation as a control loop

Automation should include sensing, deciding, acting, and validating in a closed loop.

• - Pre-checks establish safe starting state.
• - Execution applies deterministic intent.
• - Post-checks confirm policy and performance outcomes.

Scenario: Config rollout succeeds but training latency worsens

An automation job reports success across the fabric, but distributed training latency increases immediately after rollout.

Source-of-Truth
    |
Automation Pipeline
    |
Switch Fleet ---- AI Workloads

nv show interface && nv show route

Runtime state confirms whether intended change converged correctly.

iperf3 -c <peer_ip> -P 8 -t 30

Benchmark verifies regression before rollback and recovery after fix.

Scenario: Drift detected on subset of switches after emergency change

Emergency manual edits solved an incident, but drift now exists on part of the fleet.

Source-of-Truth Repo
      |
Drift Detection Job
      |
Fleet Nodes (canary + production)

Runbook: Pre-check and staged rollout

Execute a safe configuration rollout with explicit gates.

nv show interface && nv show route

Known-good baseline captured before rollout.

ansible-playbook -i inventory fabric-rollout.yml --limit canary

Canary scope converges with no execution errors.

• - Promote beyond canary only after post-check gates pass.
• - Rollback trigger should be metric-based and pre-defined.

Runbook: Drift detection and remediation

Detect divergence and safely restore intended state.

ansible-playbook -i inventory drift-check.yml

Drift list includes host, control area, and severity.

ansible-playbook -i inventory remediate-drift.yml --limit <host_or_group>

Selected nodes converge to intended configuration.

• - Always rerun validation commands after remediation.
• - Preserve audit trail linking drift to change history.

Automation success reported, but runtime state is inconsistent

Recurring drift after manual emergency changes

Walkthrough: Canary rollout with rollback gates

Execute one production-like change with deterministic safety controls.

1. Capture pre-change baseline and success thresholds.

   nv show interface && nv show route

   Expected: Baseline state and gate thresholds are recorded.

2. Apply change to canary scope only.

   ansible-playbook -i inventory fabric-rollout.yml --limit canary

   Expected: Canary converges without execution failures.

3. Run post-checks and decide promote/rollback.

   iperf3 -c <peer_ip> -P 8 -t 30

   Expected: Metrics remain within allowed tolerance.

Walkthrough: Drift remediation cycle

Detect and remediate drift while preserving workload stability.

1. Generate drift report and prioritize fixes.

   ansible-playbook -i inventory drift-check.yml

   Expected: Drift entries include severity and owner.

2. Apply targeted remediation to highest-risk nodes.

   ansible-playbook -i inventory remediate-drift.yml --limit high-risk

   Expected: High-risk nodes converge to intended state.

3. Validate policy and performance behavior post-fix.

   nv show route && iperf3 -c <peer_ip> -P 8 -t 30

   Expected: No policy/performance regressions are observed.

Lab A: Template-driven baseline deployment

Deploy standardized config from source-of-truth to multiple devices safely.

• - Render intended configuration for target scope.
• - Execute pre-checks and snapshot baseline state.
• - Apply config and verify expected state convergence.

nv show interface && nv show route

Known-good baseline captured before rollout.

Lab B: Drift detection and remediation

Detect unauthorized or accidental drift and restore intended state.

• - Run drift comparison between intent and live state.
• - Classify drift by severity and operational risk.
• - Apply corrective action and re-verify compliance.

ansible-playbook -i inventory fabric-rollout.yml --limit canary

Canary scope converges with no execution errors.

Lab C: Canary rollout with rollback guardrails

Practice low-risk staged rollout for high-impact changes.

• - Select canary subset and define success criteria.
• - Promote only if canary metrics pass thresholds.
• - Trigger rollback when gate conditions fail.

ansible-playbook -i inventory drift-check.yml

Drift list includes host, control area, and severity.

Lab D: Config change performance validation

Ensure configuration changes preserve workload SLOs.

• - Capture benchmark and telemetry baseline before change.
• - Apply one controlled change and rerun benchmark.
• - Validate throughput/latency and queue behavior post-change.

ansible-playbook -i inventory remediate-drift.yml --limit <host_or_group>

Selected nodes converge to intended configuration.