GPU and Cloud Computing

Track 1: GPU architecture and performance fundamentals

Exam questions often require translating workload behavior into hardware-aware decisions.

• - Understand compute throughput vs memory bandwidth bottlenecks at a high level.
• - Recognize when workloads are limited by kernel execution, host-device transfer, or storage ingest.
• - Use utilization telemetry to distinguish underutilization from true hardware limits.

Drill: Take one workload and classify its primary bottleneck using basic utilization and throughput metrics.

Track 2: Reproducible runtime environments

Cloud and on-prem consistency depends on explicit environment management.

• - Conda environments and lock files help keep dependency graphs reproducible across machines.
• - Container images package runtime + dependencies so deploy behavior matches test behavior.
• - CUDA, driver, and framework compatibility must be validated before benchmark conclusions.

Drill: Build one reproducible GPU environment spec and verify it runs consistently on two separate hosts.

Track 3: Multi-GPU and distributed execution patterns

Scaling beyond one GPU is core scope for accelerated data science workloads.

• - Distributed execution requires explicit worker topology and communication-aware configuration.
• - Dask-CUDA helps map workers to GPUs and exposes useful diagnostics for tuning.
• - Scaling efficiency should be measured, not assumed; communication overhead can dominate.

Drill: Run a workload on 1 GPU and multi-GPU, then report speedup efficiency and limiting factors.

Track 4: Cloud capacity, autoscaling, and operations

Production GPU workloads need repeatable capacity control and safe scale-up/scale-down behavior.

• - Choose instance families based on VRAM, interconnect, and storage/network profile.
• - Autoscaling policies should track meaningful signals (queue depth, utilization, latency SLOs).
• - Quota, placement, and startup latency constraints affect real scaling behavior.

Drill: Design a baseline autoscaling policy and define safe minimum/maximum capacity boundaries.

Track 5: Storage and data-path throughput

Data path design often determines end-to-end training or inference throughput.

• - Columnar formats and partitioning strategy influence read efficiency in distributed jobs.
• - Small-file patterns can create metadata overhead and limit effective throughput.
• - Direct high-bandwidth paths and locality-aware design reduce data-loading stalls.

Drill: Compare two dataset layout strategies and identify which one minimizes ingestion bottlenecks.

Track 6: Benchmarking and cost-performance analysis

The exam emphasizes defensible performance conclusions, not just raw runtime numbers.

• - Benchmark runs must be reproducible: fixed inputs, controlled warm-up, and comparable settings.
• - Track throughput, latency, cost, and stability together when evaluating deployment options.
• - Use standard benchmark framing (such as MLPerf-style discipline) for fair comparisons.

Drill: Write a benchmark protocol for one workload including metrics, run conditions, and pass/fail criteria.

Exam Decision Hierarchy

Prioritize decisions in this order: safety and hardware integrity, baseline consistency, controlled validation, then optimization.

• - If integrity checks fail, stop optimization and remediate first.
• - Compare against known-good baseline before changing multiple variables.
• - Document rationale for each decision to support incident replay.

Operational Evidence Standard

Treat every key action as evidence-producing: command, output, timestamp, and expected vs observed behavior.

• - Evidence should be reproducible by another engineer.
• - Use stable command templates for repeated environments.
• - Keep concise but complete validation artifacts for exam-style reasoning.

Infrastructure-first performance reasoning

GPU performance issues often originate in environment, data path, or scaling topology before model-level tuning.

• - Separate compute bottlenecks from I/O, network, and orchestration constraints.
• - Validate driver/CUDA/framework compatibility before interpreting benchmark results.
• - Treat observability baseline as a prerequisite for capacity decisions.

Scaling efficiency over raw speed

Exam decisions should prioritize efficiency and stability, not only shortest runtime on one run.

• - Measure throughput speedup and efficiency from 1 GPU to N GPUs.
• - Account for communication overhead and startup latency in cloud scenarios.
• - Compare throughput-per-dollar when choosing instance families.

Benchmark governance and reproducibility

Benchmark conclusions are only defensible when inputs, warm-up, and metrics are controlled.

• - Use fixed datasets and run protocol across all candidates.
• - Report latency percentiles, throughput, and cost together.
• - Archive benchmark config and environment metadata for replay.

Scenario A: Multi-GPU cloud deployment scales poorly from 1 to 8 GPUs

A training workload shows good single-GPU throughput but weak scaling on a multi-GPU cloud cluster. You need to identify bottlenecks and restore efficiency.

[Object Storage] -> [Data Loader] -> [GPU Workers]
                                  |
                        [Interconnect + Scheduler]
                                  |
                           [Metrics/Tracing]

nvidia-smi --query-gpu=index,name,utilization.gpu,memory.used --format=csv -l 1

index, name, utilization.gpu [%], memory.used [MiB]
0, NVIDIA A100, 91 %, 29210 MiB

Scenario B: Autoscaling reduces cost but increases tail latency

Inference service autoscaling lowered cost, but p95 latency SLO is frequently violated during traffic spikes.

[API Gateway] -> [Inference Service Pods] -> [GPU Node Group]
      |                    |
 [Queue Depth]      [HPA/Autoscaler]

Runtime and compatibility verification

Confirm node runtime is valid before benchmark or scaling analysis.

nvidia-smi

Shows GPU inventory, driver version, CUDA compatibility, utilization table.

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Container reports expected GPU list and compatible driver runtime.

• - Do this before tuning to avoid wasting time on invalid runtime stacks.
• - Capture command output in benchmark metadata.

Scaling and cost-performance runbook

Run controlled scaling measurements and compute efficiency metrics.

python - <<'PY'
for gpus in [1,2,4,8]:
    # run workload with fixed input and warm-up
    print(gpus, 'throughput=', '...')
PY

1 throughput=...
2 throughput=...
4 throughput=...

python - <<'PY'
base=1000
for gpus,thr,cost in [(1,1000,1.0),(2,1800,2.0),(4,3200,4.2)]:
    eff=thr/(base*gpus)
    tpd=thr/cost
    print(gpus, round(eff,2), round(tpd,2))
PY

1 1.0 1000.0
2 0.9 900.0
4 0.8 761.9

• - Always use fixed data snapshot and warm-up policy for fair comparison.
• - Interpret efficiency and economics together.

Benchmark results are inconsistent across runs

Autoscaling meets average latency but fails p95 SLO

Walkthrough A: Build a reproducible GPU benchmark harness

Create a benchmark protocol that supports fair cloud and hardware comparisons.

1. Validate runtime compatibility on each candidate node.

   nvidia-smi

   Expected: Driver/CUDA stack is valid and consistent with workload runtime requirements.

2. Run benchmark with fixed warm-up and fixed measurement window.

   Expected: You capture comparable throughput/latency across candidates.

3. Compute throughput-per-dollar and scaling efficiency summary.

   Expected: Decision table ranks candidates by both performance and economics.

Walkthrough B: Diagnose sublinear multi-GPU scaling

Find root causes when scaling from one GPU to many yields poor efficiency.

1. Collect utilization and step-time baseline on 1 GPU and N GPUs.

   Expected: You have direct evidence of where scaling loss appears.

2. Inspect data-loader and communication overhead for bottlenecks.

   Expected: Primary bottleneck is classified as I/O, comms, or scheduling.

3. Apply one targeted change and rerun controlled benchmark.

   Expected: Scaling efficiency improves with measurable evidence.

Lab A: Environment reproducibility

Package a GPU workflow so results are reproducible across systems.

• - Create environment specification and container runtime setup.
• - Run identical benchmark command on two hosts or environments.
• - Document any compatibility mismatch and the fix.

nvidia-smi

Shows GPU inventory, driver version, CUDA compatibility, utilization table.

Lab B: Scaling-efficiency lab

Measure real speedup and identify communication or scheduling bottlenecks.

• - Run baseline on one GPU and expanded run on multiple GPUs.
• - Compute throughput speedup and scaling efficiency.
• - List top factors reducing ideal linear scaling.

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

Container reports expected GPU list and compatible driver runtime.

Lab C: Cloud capacity planning

Select cloud configuration that balances cost, reliability, and performance.

• - Compare at least two candidate instance families for workload fit.
• - Define autoscaling signal(s), floor/ceiling, and cooldown behavior.
• - Write failover or fallback plan for capacity shortages.

python - <<'PY'
for gpus in [1,2,4,8]:
    # run workload with fixed input and warm-up
    print(gpus, 'throughput=', '...')
PY

1 throughput=...
2 throughput=...
4 throughput=...

Lab D: Storage path optimization

Reduce ingestion stalls by improving data layout and access path.

• - Benchmark one small-file-heavy layout and one consolidated layout.
• - Record impact on startup latency and steady-state throughput.
• - Choose a storage and partitioning strategy for production-like runs.

python - <<'PY'
base=1000
for gpus,thr,cost in [(1,1000,1.0),(2,1800,2.0),(4,3200,4.2)]:
    eff=thr/(base*gpus)
    tpd=thr/cost
    print(gpus, round(eff,2), round(tpd,2))
PY

1 1.0 1000.0
2 0.9 900.0
4 0.8 761.9