Troubleshooting Tools

Track 1: Layered troubleshooting model

High-severity incidents are resolved faster when symptoms are classified by layer before tuning.

• - Start with symptom classification: host, switch, fabric, or policy.
• - Use deterministic command order and timestamp evidence.
• - Correlate telemetry with workload windows before declaring root cause.

Drill: Build a one-page triage flow from application symptom to network-layer candidate.

Track 2: Switch-side diagnostics (Cumulus CLI and BERT)

Switch health and link-quality evidence often reveals root cause before host changes are attempted.

• - Validate interface states, counters, errors, and queue pressure first.
• - Use BERT and optics/link diagnostics to isolate physical-layer instability.
• - Confirm route/policy correctness before performance tuning.

Drill: Collect switch evidence for one degraded link and propose first remediation step.

Track 3: Host-side diagnostics and end-to-end path checks

Host NIC/runtime conditions can mimic fabric-level failures.

• - Use host tools to verify NIC health, drops, offload state, and interface errors.
• - Confirm endpoint reachability and throughput from both ends of a path.
• - Separate host-local bottlenecks from shared-fabric congestion.

Drill: Run host diagnostics on two nodes and identify whether issue is local or network-wide.

Track 4: Packet and benchmark evidence

Packet captures and benchmark tools provide objective proof, not assumptions.

• - Use tcpdump and counters to confirm packet loss, retransmit, or policy drops.
• - Use repeatable benchmark windows for before/after remediation validation.
• - Track latency percentile and throughput changes together.

Drill: Capture and compare packet and benchmark evidence before and after one change.

Track 5: Congestion and bottleneck troubleshooting

Bottleneck and congestion scenarios are explicit blueprint expectations.

• - Identify sustained hot links versus short burst events.
• - Apply one controlled change at a time and validate impact.
• - Use rollback-safe plans when tuning buffer/queue behavior.

Drill: Design a congestion remediation plan with measurable acceptance criteria.

Exam Decision Hierarchy

Prioritize decisions in this order: safety and hardware integrity, baseline consistency, controlled validation, then optimization.

• - If integrity checks fail, stop optimization and remediate first.
• - Compare against known-good baseline before changing multiple variables.
• - Document rationale for each decision to support incident replay.

Operational Evidence Standard

Treat every key action as evidence-producing: command, output, timestamp, and expected vs observed behavior.

• - Evidence should be reproducible by another engineer.
• - Use stable command templates for repeated environments.
• - Keep concise but complete validation artifacts for exam-style reasoning.

Evidence hierarchy in incident response

Symptoms become actionable only when anchored to objective evidence from multiple layers.

• - Correlate workload behavior with network counters and packet data.
• - Capture command output with timestamps for replayability.
• - Prioritize reversible actions during uncertainty.

Switch and host parity checks

Cross-validating switch and host views prevents false conclusions from one-sided telemetry.

• - If counters disagree, verify collection windows and interface mapping.
• - Host-local defects can present as network-wide symptoms.
• - Consistent two-sided checks reduce misdiagnosis risk.

Troubleshooting as controlled experimentation

Each remediation is an experiment with a hypothesis, action, and measurable outcome.

• - Define expected change before execution.
• - Reject fixes without repeatable improvement.
• - Keep rollback trigger criteria explicit.

Scenario: Throughput drops 25% during nightly distributed training

Training jobs started failing SLO after a maintenance window; throughput dropped and latency spikes increased.

Training Nodes
   |
Leaf-Spine Fabric
   |
Storage and Services

nv show interface && nv show interface counters

Counter trends identify whether errors/drops are localized.

ethtool -S <nic> | egrep 'drop|error|timeout'

Host-side drops/errors confirm or reject endpoint-local cause.

Scenario: Intermittent packet loss appears after tenant policy change

A security policy update introduced intermittent communication failure for one tenant during peak hours.

Tenant A/B Workloads
     |
Policy Controls + Fabric
     |
Shared Services

Runbook: Switch and host baseline collection

Collect deterministic pre-change evidence for high-confidence root-cause analysis.

nv show interface && nv show route

State matches expected topology and policy intent.

ip -s link show <nic> && ethtool -S <nic>

Error/drop patterns are identified with clear directionality.

• - Always compare to known-good baseline or previous window.
• - Record both source and destination endpoint evidence.

Runbook: Packet and benchmark validation

Prove remediation with packet-level evidence and workload impact checks.

sudo tcpdump -i <nic> host <peer_ip> -w incident-window.pcap

Capture supports packet-loss/retransmit or policy-drop analysis.

iperf3 -c <peer_ip> -P 8 -t 30

Throughput and stability align with target baseline range.

• - Use the same benchmark profile for before/after comparison.
• - Store pcap and benchmark logs with incident timestamps.

False root cause due to single-layer troubleshooting

Congestion returns after temporary mitigation

Walkthrough: End-to-end troubleshooting evidence capture

Build complete evidence package for a network performance regression.

1. Capture baseline switch and host counters.

   nv show interface counters && ip -s link show <nic>

   Expected: Pre-change evidence captured with timestamps.

2. Capture packet sample during incident window.

   sudo tcpdump -i <nic> -w pre-fix.pcap

   Expected: Packet file includes representative failure window.

3. Apply one scoped remediation and rerun benchmark.

   iperf3 -c <peer_ip> -P 8 -t 30

   Expected: Post-fix benchmark shows measurable improvement.

Walkthrough: Congestion hotspot isolation

Identify and remediate one sustained congestion hotspot safely.

1. Locate persistent high-pressure interfaces.

   nv show interface counters

   Expected: Hotspot links are clearly identified.

2. Validate endpoint behavior and route consistency.

   ip route show && ethtool -S <nic>

   Expected: Endpoint and path evidence supports congestion hypothesis.

3. Apply one remediation and revalidate over repeated windows.

   Expected: Queue pressure and latency variance reduce sustainably.

Lab A: Switch-first incident triage

Isolate whether issue originates in interface/link/queue state.

• - Capture baseline switch interface and error counters.
• - Collect queue/utilization evidence during workload peak.
• - Classify issue as physical, route/policy, or congestion-related.

nv show interface && nv show route

State matches expected topology and policy intent.

Lab B: Host and path diagnostics

Differentiate host-local NIC issues from shared network faults.

• - Run host NIC and interface checks on source and destination nodes.
• - Execute controlled path validation tests.
• - Correlate endpoint evidence with switch telemetry.

ip -s link show <nic> && ethtool -S <nic>

Error/drop patterns are identified with clear directionality.

Lab C: Packet evidence and policy validation

Use packet capture and counters to prove drop cause.

• - Capture traffic at host edge during failure window.
• - Map packet behavior to policy and route expectations.
• - Document root cause with packet + counter evidence.

sudo tcpdump -i <nic> host <peer_ip> -w incident-window.pcap

Capture supports packet-loss/retransmit or policy-drop analysis.

Lab D: Congestion remediation validation

Apply one tuning change and confirm measurable improvement.

• - Define baseline latency/throughput/counter metrics.
• - Apply one targeted remediation and rerun workload.
• - Validate improvement and absence of side effects.

iperf3 -c <peer_ip> -P 8 -t 30

Throughput and stability align with target baseline range.