Exam focus
Primary domain: Productionizing LLM Solutions (22%).
Inference Optimization
- Quantization (INT8, FP16)
- Pruning
- Knowledge distillation
- TensorRT optimization
- Batch inference
- Streaming inference
- KV caching
- Model compression
Scaling & Infrastructure
- GPU acceleration
- CUDA basics
- Multi-GPU scaling
- Distributed inference
- Throughput vs latency tradeoff
- Autoscaling
NVIDIA Ecosystem
- NVIDIA NeMo
- NVIDIA Triton Inference Server
- NVIDIA NIM
- CUDA
- TensorRT
- DGX systems
- RAPIDS (cuDF awareness)
- GPU memory management
Scope Bullet Explanations
- Quantization (INT8, FP16): Reduces numerical precision to improve speed and memory efficiency.
- Pruning: Removes less-contributive weights/channels to shrink model cost.
- Knowledge distillation: Trains a smaller student model using a larger teacher’s behavior.
- TensorRT optimization: NVIDIA runtime/compiler optimizations for fast GPU inference.
- Batch inference: Processes multiple requests together to increase throughput.
- Streaming inference: Returns tokens incrementally for lower perceived latency.
- KV caching: Reuses prior attention keys/values during autoregressive decoding.
- Model compression: Broad set of techniques to reduce model size and serving cost.
- GPU acceleration: Parallel compute on GPUs for high-throughput LLM workloads.
- CUDA basics: NVIDIA parallel programming/runtime model for GPU computation.
- Multi-GPU scaling: Distributes inference/training across multiple GPUs.
- Distributed inference: Serving architecture where model execution spans multiple nodes/devices.
- Throughput vs latency tradeoff: Higher batching/utilization can increase queueing delay.
- Autoscaling: Dynamically adjusts serving capacity based on demand signals.
- NVIDIA NeMo: NVIDIA framework ecosystem for model development/adaptation workflows.
- NVIDIA Triton Inference Server: Production model-serving platform with multi-backend support.
- NVIDIA NIM: Packaged inference microservices for simplified deployment.
- CUDA: Core software layer enabling GPU compute workloads.
- TensorRT: Inference optimization stack for NVIDIA hardware.
- DGX systems: NVIDIA integrated AI infrastructure platforms for high-performance workloads.
- RAPIDS (cuDF awareness): GPU-accelerated data processing ecosystem relevant to ML pipelines.
- GPU memory management: Strategies to handle VRAM allocation, fragmentation, and stability.
Chapter overview
Production success is determined by serving architecture, optimization strategy, and hardware-software integration. This chapter focuses on inference performance, scaling tradeoffs, and NVIDIA platform components commonly used in enterprise LLM stacks.
Learning objectives
- Apply quantization, pruning, distillation, and TensorRT optimization in deployment planning.
- Balance throughput and latency under real traffic conditions.
- Understand multi-GPU and distributed inference design principles.
- Map roles of NeMo, Triton, NIM, CUDA, TensorRT, DGX, and RAPIDS awareness.
11.1 Inference optimization toolkit
Quantization
INT8 or lower-precision variants can reduce memory and increase speed. Quality impact depends on model architecture, calibration, and workload.
Pruning and distillation
Pruning removes less important parameters; distillation transfers behavior from large teacher to smaller student. Both target efficiency with acceptable quality loss.
TensorRT
TensorRT optimizes graphs and kernels for NVIDIA GPUs. Typical gains include reduced latency and improved throughput for stable serving workloads.
Batch and streaming inference
- Batch inference improves throughput via amortized compute.
- Streaming inference improves responsiveness for interactive applications. Your serving mode should match user experience requirements.
KV caching and compression
KV cache reduces repeated attention computation during generation. Compression and memory strategies can improve capacity under high concurrency.
11.2 Infrastructure scaling
GPU acceleration and CUDA basics
CUDA enables large-scale parallel compute on NVIDIA GPUs. Understanding kernel execution and memory behavior helps explain performance bottlenecks.
GPU vs CPU inference considerations
- GPUs are typically preferred for high-throughput, low-latency LLM inference due to massive parallelism.
- CPUs can be sufficient for small models, low-concurrency workloads, and cost-sensitive edge or control-plane tasks.
- Decision criteria should include latency target, request concurrency, model size, memory bandwidth, and total cost per successful request.
- Hybrid strategies are common: CPU for orchestration and lightweight preprocessing, GPU for core model execution.
Multi-GPU and distributed inference
Scale-out options include tensor parallelism, pipeline parallelism, and replica-based serving. Choose based on model size and latency goals.
Throughput vs latency
High batch sizes improve utilization but may increase queue delay. SLO-driven tuning should prioritize user-visible latency targets first.
Autoscaling
Autoscaling adds capacity for burst traffic. Effective policies require predictive signals and warm-start behavior to avoid cold-start penalties.
11.3 NVIDIA stack mapping
NeMo
Framework for model customization, training, and adaptation workflows.
Triton Inference Server
Production-grade multi-model serving and backend orchestration.
NIM
Packaged inference microservices to simplify deployment and operational consistency.
CUDA, TensorRT, DGX
Core acceleration stack from programming model to optimized runtime and integrated hardware systems.
RAPIDS (cuDF awareness)
Useful in adjacent data-processing pipelines where GPU-accelerated data workflows support LLM applications.
11.4 Capacity planning and observability
Track:
- p50/p95/p99 latency,
- tokens/sec,
- GPU memory utilization,
- admission queue depth,
- error rates and timeout rates,
- cost per successful request. Integrate telemetry with automated scaling and incident response.
11.5 Failure modes
- Over-quantization causing unacceptable quality drop.
- Throughput tuning that violates interactive latency SLOs.
- Ignoring GPU memory fragmentation and OOM patterns.
- Deploying optimized engines without regression test baselines.
Chapter summary
Deployment quality requires deliberate optimization and platform-aware design. NVIDIA stack components are most effective when integrated with clear SLOs, telemetry, and rollback controls.
Mini-lab: deployment optimization trial
Goal: compare two serving configurations.
- Establish baseline model serving in FP16.
- Build quantized or TensorRT-optimized variant.
- Run identical load profile on both.
- Compare latency, throughput, and quality metrics.
- Choose production candidate and rollback trigger. Deliverable in Notion:
- Deployment comparison report with final recommendation.
Review questions
- When does quantization provide highest ROI?
- What risks come with aggressive quantization?
- How does TensorRT improve inference performance?
- Why can batching hurt interactive user experience?
- What signals indicate GPU memory bottlenecks?
- How do Triton and NIM differ in operational role?
- What is the relationship between autoscaling and cold-start latency?
- Why should p95 and p99 be tracked in addition to averages?
- How does KV caching affect long-response workloads?
- What minimum telemetry is needed before production launch?
- In which scenarios is CPU inference still a valid choice over GPU inference?
Key terms
Quantization, pruning, distillation, TensorRT, Triton, NIM, CUDA, DGX, GPU vs CPU inference tradeoff, KV cache, autoscaling, p95 latency, throughput.
Exam traps
- Optimizing benchmark throughput while missing live SLOs.
- Treating one optimized configuration as universal across workloads.
- Underestimating memory behavior in multi-tenant serving.