MLOps

Track 1: Reproducible model packaging and versioning

Operational success begins with reproducible artifacts and traceable versions.

• - Bundle model, preprocessing contract, and runtime dependencies together.
• - Track model, data, and configuration versions so outputs remain auditable.
• - Use explicit artifact promotion stages (dev, staging, prod).

Drill: Create a release artifact checklist covering model weights, preprocessing schema, and runtime metadata.

Track 2: Triton serving fundamentals

NCP-ADS scope explicitly includes Triton-oriented deployment concepts.

• - Understand Triton model repository layout and backend choices.
• - Configure model-specific inference settings via model configuration files.
• - Use deployment patterns that allow safe updates without service disruption.

Drill: Stand up one Triton-served model and validate request/response behavior with defined input and output contracts.

Track 3: Inference performance tuning

Serving quality depends on balancing throughput, latency, and resource limits.

• - Dynamic batching and instance-group sizing change latency-throughput tradeoffs.
• - Response caching can reduce repeated inference cost for repeated queries.
• - Optimization decisions must be validated under representative traffic, not synthetic micro-tests only.

Drill: Run two serving configurations and compare p95 latency, throughput, and GPU utilization.

Track 4: Observability and model monitoring

You cannot operate ML services safely without system and model-level telemetry.

• - Track service metrics (latency, throughput, errors) and model metrics (drift, quality proxy signals).
• - Use dashboards and alert thresholds tied to SLO and incident response expectations.
• - Separate infrastructure failures from data or model behavior changes during triage.

Drill: Define one monitoring dashboard with service, data, and model health sections plus alert thresholds.

Track 5: Release strategies and rollback

Controlled rollouts reduce blast radius when model behavior changes unexpectedly.

• - Canary and shadow deployments provide early safety signals before full rollout.
• - Rollback criteria must be explicit and connected to monitored metrics.
• - Release notes should document feature, data, and configuration changes.

Drill: Create a rollout runbook with canary gates, rollback triggers, and on-call ownership.

Track 6: Security, governance, and compliance

Production ML systems must address access control, data protection, and operational governance.

• - Apply least-privilege access to model repositories, serving endpoints, and secrets.
• - Protect sensitive data and model artifacts in transit and at rest.
• - Audit logs and version history support compliance and incident forensics.

Drill: Perform a lightweight security review for one inference service and capture remediation actions.

Exam Decision Hierarchy

Prioritize decisions in this order: safety and hardware integrity, baseline consistency, controlled validation, then optimization.

• - If integrity checks fail, stop optimization and remediate first.
• - Compare against known-good baseline before changing multiple variables.
• - Document rationale for each decision to support incident replay.

Operational Evidence Standard

Treat every key action as evidence-producing: command, output, timestamp, and expected vs observed behavior.

• - Evidence should be reproducible by another engineer.
• - Use stable command templates for repeated environments.
• - Keep concise but complete validation artifacts for exam-style reasoning.

Train-serve contract integrity

Serving reliability depends on keeping preprocessing, schema, and model versions synchronized across training and inference.

• - Package preprocessing contract with model artifact, not separately in tribal knowledge.
• - Version model, data snapshot, and runtime configuration together.
• - Validate contract parity in staging before rollout.

SLO-driven serving optimization

Inference tuning should optimize toward target SLO (for example p95 latency) rather than raw throughput only.

• - Dynamic batching and instance groups trade latency and throughput differently.
• - Traffic shape and request size mix matter more than microbenchmark best case.
• - Use representative load tests and canary signals for final config choice.

Release governance and rollback readiness

MLOps maturity is demonstrated by safe rollout strategy and fast rollback criteria tied to observable signals.

• - Define canary gates before deployment starts.
• - Preserve previous model version for rapid fallback.
• - Tie rollback triggers to latency, error, and model-quality indicators.

Scenario A: New model deployment increases p95 latency and error spikes

A newly deployed Triton model version passes smoke tests but violates latency SLO and raises error rate in production canary traffic.

[Client] -> [Gateway] -> [Triton Inference Service]
                          |
                     [Model Repository]
                          |
                 [Metrics + Alerting]

curl -s http://localhost:8002/metrics | grep -E 'nv_inference|request_duration' | head

nv_inference_request_success ...
nv_inference_queue_duration_us ...

Scenario B: Offline metrics strong but online quality drifts after release

Model performs well offline, but online proxy metrics indicate drift and user complaints increase.

[Feature Stream] -> [Inference Endpoint] -> [Prediction Logs]
        |                                  |
 [Drift/Proxy Checks]               [Alert + Triage]

Triton deployment sanity runbook

Validate model repository and serving endpoint behavior before scaling traffic.

tritonserver --model-repository=/models

Triton server starts and reports loaded model versions.

curl -s http://localhost:8000/v2/health/ready && echo

Returns HTTP 200 with readiness status.

• - Run readiness checks before any performance test.
• - Capture server logs and loaded model list in release artifact.

Serving performance and rollback runbook

Measure SLO metrics under load and execute rollback when gates fail.

perf_analyzer -m my_model --concurrency-range 1:16 --measurement-interval 5000 --percentile=95

Reports throughput and latency percentiles for each concurrency level.

curl -s http://localhost:8000/v2/models/my_model/config

Shows active model config/version after rollback action.

• - Define fail gates before running perf_analyzer.
• - Rollback should be scripted to avoid delay during incidents.

Train-serve mismatch from unversioned preprocessing

Latency SLO violations after tuning for throughput only

Walkthrough A: Deploy and validate a Triton model end-to-end

Stand up model serving with health, schema, and baseline performance checks.

1. Place model in repository structure and start Triton server.

   tritonserver --model-repository=/models

   Expected: Server logs show model loaded without configuration errors.

2. Run readiness and sample inference checks.

   curl -s http://localhost:8000/v2/health/ready

   Expected: Endpoint returns ready status and sample inference succeeds.

3. Capture baseline latency and throughput with perf analyzer.

   Expected: Baseline SLO metrics are documented for future change comparisons.

Walkthrough B: Canary rollout with rollback drill

Practice controlled release with objective rollback triggers.

1. Route small traffic slice to candidate model and monitor canary gates.

   Expected: Canary health metrics are visible in near real time.

2. Trigger rollback when threshold breach is detected.

   Expected: Traffic shifts back to stable model quickly with minimal impact.

3. Publish incident summary with config and metric deltas.

   Expected: Team has concrete evidence and next action list.

Lab A: Triton deployment baseline

Deploy one model end-to-end using Triton repository conventions.

• - Package model artifact and configuration into repository structure.
• - Run local or staged Triton server and validate inference contract.
• - Capture baseline latency and throughput metrics.

tritonserver --model-repository=/models

Triton server starts and reports loaded model versions.

Lab B: Inference optimization lab

Tune serving configuration for target latency-throughput profile.

• - Compare at least two dynamic batching or instance-group settings.
• - Measure p50/p95 latency, throughput, and GPU utilization.
• - Choose configuration based on explicit SLO objective.

curl -s http://localhost:8000/v2/health/ready && echo

Returns HTTP 200 with readiness status.

Lab C: Monitoring and drift readiness

Operationalize monitoring for both service and model behavior.

• - Define alert thresholds for latency, error rate, and saturation.
• - Add model-quality or drift proxy checks where labels are delayed.
• - Test alert routing and runbook links.

perf_analyzer -m my_model --concurrency-range 1:16 --measurement-interval 5000 --percentile=95

Reports throughput and latency percentiles for each concurrency level.

Lab D: Safe rollout and rollback

Practice low-risk production change management.

• - Design canary criteria for automated or manual promotion.
• - Trigger a controlled rollback on defined failure signal.
• - Write post-release validation checklist.

curl -s http://localhost:8000/v2/models/my_model/config

Shows active model config/version after rollback action.