Machine Learning

Track 1: Parallelism strategy selection

Exam prompts often test whether you can pick the right scaling strategy for memory and throughput constraints.

• - Data Parallel (DDP) is usually the baseline for models that fit on one GPU per replica.
• - Tensor Parallel and Pipeline Parallel are used when model size or layer size exceeds single GPU limits.
• - ZeRO-style sharding reduces optimizer and state memory pressure and can be combined with other techniques.

Drill: Given three scenarios (model fits, model barely fits, largest layer does not fit), map each to the best parallelism strategy and justify tradeoffs.

Track 2: Mixed precision and Tensor Core efficiency

NCP-ADS machine learning scope is strongly GPU-centric and expects precision-aware optimization decisions.

• - Mixed precision combines lower precision compute with selective FP32 stability steps.
• - Loss scaling helps preserve small gradients and avoid underflow in reduced precision training.
• - Tensor Core throughput benefits are maximized when dimensions and batch sizes satisfy hardware-friendly multiples.

Drill: Run one training job in fp32, then mixed precision with proper loss scaling, and compare step time and memory utilization.

Track 3: Throughput and memory tuning on single GPU

Practical exam cases can start on single GPU before escalating to multi-GPU.

• - Batch size choice is the first lever; maximize useful utilization without destabilizing convergence.
• - Gradient accumulation increases effective batch size but adds extra forward/backward overhead.
• - Gradient checkpointing lowers activation memory at the cost of additional recomputation.

Drill: Tune one workload with baseline, gradient accumulation, and checkpointing; document tokens/s or samples/s and memory footprint.

Track 4: Profiling and bottleneck diagnosis

Knowing optimization tools is part of software literacy for accelerated machine learning.

• - Start with nvidia-smi and utilization metrics to identify obvious underutilization.
• - Use profiler traces to isolate time-heavy kernels and non-Tensor-Core bottlenecks.
• - Optimization should be evidence-driven: batch size, precision modes, and kernel hotspots.

Drill: Capture one profile pass, rank top kernels by time, and produce a prioritized optimization action list.

Track 5: Data quality, splitting, and generalization

Model quality failures are commonly caused by split leakage, imbalance, and unstable preprocessing.

• - Stratified splitting helps preserve target distribution across train/validation/test.
• - Feature scaling and normalization choices influence gradient-based and distance-based methods.
• - Overfitting and underfitting should be diagnosed from train/validation behavior, not a single metric snapshot.

Drill: Evaluate one dataset with random split vs stratified split and compare validation stability and minority-class behavior.

Track 6: Hyperparameter optimization workflow

The exam may include strategy-level HPO decisions rather than only algorithm internals.

• - Random search scales well in parallel; Bayesian search is sequential but increasingly informed.
• - Hyperband-style early stopping can cut compute cost for large search spaces.
• - Search-space sizing and scale choice (linear vs log) can dominate tuning efficiency.

Drill: Design an HPO plan with explicit search ranges, scaling assumptions, and parallel job limits for a constrained GPU budget.

Exam Decision Hierarchy

Prioritize decisions in this order: safety and hardware integrity, baseline consistency, controlled validation, then optimization.

• - If integrity checks fail, stop optimization and remediate first.
• - Compare against known-good baseline before changing multiple variables.
• - Document rationale for each decision to support incident replay.

Operational Evidence Standard

Treat every key action as evidence-producing: command, output, timestamp, and expected vs observed behavior.

• - Evidence should be reproducible by another engineer.
• - Use stable command templates for repeated environments.
• - Keep concise but complete validation artifacts for exam-style reasoning.

Parallelism decision framework

Choose scaling mode based on fit constraints first, then optimize communication and efficiency.

• - If model fits on one GPU, start with DDP baseline and measure scaling.
• - If layer/model does not fit, evaluate TP/PP or ZeRO-style memory sharding.
• - Use topology and interconnect awareness when interpreting scaling results.

Precision and stability engineering

Mixed precision boosts throughput, but stability checks and loss-scaling controls are required for safe training.

• - Compare fp32 and mixed precision with fixed seed and identical data path.
• - Monitor overflow/underflow signals and convergence behavior.
• - Report throughput and validation quality together.

Profiler-driven optimization loop

Optimization should follow measured bottlenecks: input pipeline, kernels, memory, or communication.

• - Capture a targeted profiling window after warm-up.
• - Rank hotspots by contribution to total step time.
• - Apply one change at a time and validate improvement.

Scenario A: Mixed precision improves speed but validation quality regresses

After enabling mixed precision, training throughput increases but validation accuracy drops unexpectedly.

[Data Loader] -> [Forward/Backward] -> [Optimizer Step]
                     |
             [Precision + Loss Scaling]

python train.py --precision bf16 --grad-accum 1 --log-interval 50

step=500 loss=... throughput=... tokens/s

Scenario B: DDP scale-out from 1 to 8 GPUs gives weak speedup

Training scales poorly despite additional GPUs. You need to classify whether bottleneck is data, communication, or kernel efficiency.

[Sharded Dataset] -> [GPU Worker x N] -> [AllReduce]
          |                              |
     [Input Pipeline]               [Interconnect]

Training baseline and precision comparison

Establish reproducible baseline before advanced scaling changes.

python train.py --precision fp32 --devices 1 --seed 42

epoch=1 step_time=... val_metric=...

python train.py --precision bf16 --devices 1 --seed 42

epoch=1 step_time=... val_metric=...

• - Do not change dataset, seed, or augmentation settings between comparisons.
• - Track both throughput and validation metric delta.

Distributed scaling and profiling runbook

Measure DDP efficiency and inspect bottlenecks with profiler tooling.

for n in 1 2 4 8; do torchrun --nproc_per_node=$n train.py --precision bf16 --seed 42; done

n=1 ... samples/s
n=2 ...
n=4 ...

nsys profile -o train_profile --trace=cuda,nvtx,osrt python train.py --max-steps 200

Profile report generated: train_profile.qdrep

• - Use short profiling windows after warm-up to reduce trace noise.
• - Re-run only one tuned parameter at a time after profile analysis.

OOM and unstable throughput when increasing effective batch size

High training score but weak generalization

Walkthrough A: Precision and memory tradeoff benchmark

Compare fp32 and mixed precision with stable, reproducible setup.

1. Run fp32 baseline and capture throughput, memory, and validation metrics.

   python train.py --precision fp32 --devices 1 --seed 42

   Expected: Baseline metrics are stored for direct comparison.

2. Run mixed precision with identical data and seed settings.

   python train.py --precision bf16 --devices 1 --seed 42

   Expected: Throughput and memory change is measured without confounding factors.

3. Decide deployment precision policy with quality guardrail.

   Expected: Final decision includes both speed gain and validation tolerance.

Walkthrough B: Distributed scaling diagnosis lab

Investigate why scaling efficiency drops as GPU count increases.

1. Run scaling sweep at 1, 2, and 4+ GPUs and capture throughput.

   Expected: Efficiency curve reveals where scaling starts to degrade.

2. Profile one degraded run for communication and input stalls.

   Expected: Primary bottleneck category is identified with evidence.

3. Apply one targeted change and rerun sweep.

   Expected: Efficiency improves or limitation is clearly documented.

Lab A: Precision and throughput

Quantify impact of mixed precision and TF32/BF16 options on performance and accuracy.

• - Establish fp32 baseline with fixed seed and data pipeline.
• - Enable mixed precision path and compare wall-clock step time.
• - Confirm no unacceptable quality regression on validation metrics.

python train.py --precision fp32 --devices 1 --seed 42

epoch=1 step_time=... val_metric=...

Lab B: Multi-GPU strategy fit

Choose and validate a scaling strategy for model-size and interconnect constraints.

• - Classify workload into fit/no-fit/layer-too-large category.
• - Prototype DDP and one memory-sharding method where relevant.
• - Record communication overhead observations and effective speedup.

python train.py --precision bf16 --devices 1 --seed 42

epoch=1 step_time=... val_metric=...

Lab C: Profiling-led optimization

Use profiler evidence to improve utilization.

• - Capture one targeted profiling window after warm-up iterations.
• - Identify top kernels and non-compute stalls.
• - Apply two targeted changes and verify measurable improvement.

for n in 1 2 4 8; do torchrun --nproc_per_node=$n train.py --precision bf16 --seed 42; done

n=1 ... samples/s
n=2 ...
n=4 ...

Lab D: Generalization and HPO discipline

Improve model robustness while controlling tuning cost.

• - Use stratified splits for imbalanced targets.
• - Run bounded HPO with explicit scaling and parallel-job limits.
• - Explain why chosen configuration generalizes beyond training metrics.

nsys profile -o train_profile --trace=cuda,nvtx,osrt python train.py --max-steps 200

Profile report generated: train_profile.qdrep