Administration

Track 1: OS and node lifecycle management

Stable operations depend on disciplined patching, drift control, and host baseline integrity.

• - Define maintenance windows and change validation criteria.
• - Track version drift across control and worker nodes.
• - Verify post-maintenance node and runtime health.

Drill: Create an OS maintenance checklist with pre/post validation commands.

Track 2: Scheduler administration

Workload reliability depends on correct scheduler health, policies, and queue behavior.

• - Administer Kubernetes and scheduler state with evidence-based checks.
• - Validate policy changes through controlled test workloads.
• - Keep scheduler health dashboards aligned to SLOs.

Drill: Run a scheduler health audit and identify one latent risk before incident occurs.

Track 3: User, role, and quota governance

Misconfigured RBAC/quotas can cause both security and capacity incidents.

• - Map users and roles to least-privilege policy.
• - Set and validate resource quotas per project/tenant.
• - Audit governance changes with traceability.

Drill: Design a quota and role matrix for two teams with different priority levels.

Track 4: Data/model management

Operational quality requires repeatable handling of model artifacts and dataset versions.

• - Track model and dataset versions used by production workloads.
• - Validate artifact integrity and access policies.
• - Align lifecycle procedures with rollback/recovery needs.

Drill: Build a model promotion checklist including version, access, and rollback fields.

Track 5: NGC private registry administration

Registry configuration and credentials are central dependencies for controlled deployments.

• - Validate API key and registry connectivity from target environments.
• - Enforce access policy boundaries for teams and projects.
• - Audit registry usage for dependency and compliance visibility.

Drill: Simulate API key rotation and verify zero-downtime artifact access.

Exam Decision Hierarchy

Prioritize decisions in this order: safety and hardware integrity, baseline consistency, controlled validation, then optimization.

• - If integrity checks fail, stop optimization and remediate first.
• - Compare against known-good baseline before changing multiple variables.
• - Document rationale for each decision to support incident replay.

Operational Evidence Standard

Treat every key action as evidence-producing: command, output, timestamp, and expected vs observed behavior.

• - Evidence should be reproducible by another engineer.
• - Use stable command templates for repeated environments.
• - Keep concise but complete validation artifacts for exam-style reasoning.

Administration as control plane hygiene

Admin operations maintain platform hygiene so workloads run predictably and securely.

• - Treat drift detection as continuous work.
• - Codify recurring tasks into runbooks.
• - Measure admin effectiveness with incident reduction metrics.

Governance and capacity are coupled

Role and quota policy decisions influence both security posture and workload fairness.

• - Quota policy should align with priority and business impact.
• - RBAC changes should be tested before full rollout.
• - Audit trails are mandatory for high-stakes environments.

Artifact lifecycle as operational dependency

Model and dataset lifecycle management is core operations work, not a one-time release task.

• - Track provenance for every promoted artifact.
• - Validate access policy and integrity at each stage.
• - Pre-plan rollback path before promotion.

Scenario: Quota policy change causes high-priority job starvation

A new quota policy was deployed and high-priority jobs are now queued while low-priority workloads consume resources.

Users/Teams
   |
RBAC + Quotas
   |
Scheduler
   |
GPU Worker Pool

kubectl get resourcequota -A

Quota values match intended project capacity policy.

sacctmgr show assoc

Associations reflect expected account and limit settings.

Scenario: Registry credential update breaks model pulls

After API key rotation, some workloads cannot pull model artifacts from NGC private registry.

NGC API Key
    |
Registry Access Policy
    |
Runtime Artifact Pull

Administration baseline runbook

Capture core admin state before and after maintenance or policy changes.

kubectl get nodes -o wide

Nodes report Ready and expected software/runtime profile.

squeue

Queue status reflects expected workload distribution and priorities.

cmsh -c 'show version'

BCM reports expected installed version and CLI access.

• - Capture before and after any admin maintenance event.
• - Store outputs with change-ticket identifiers.

Governance and registry verification runbook

Validate RBAC/quota governance and registry access posture.

kubectl get rolebinding,clusterrolebinding -A

Bindings align with approved least-privilege model.

kubectl get resourcequota -A

Resource quotas reflect approved project allocations.

ngc config current

NGC config reports expected org/team and key context.

• - Pair access checks with policy approval records.
• - Revalidate immediately after key rotation events.

Silent drift in scheduler configuration

Quota misalignment causes team friction and missed SLAs

Registry access failures after credential changes

Walkthrough: Scheduler and governance health audit

Verify scheduler state, role bindings, and quotas in one repeatable admin pass.

1. Collect cluster and queue baseline.

   kubectl get nodes -o wide && squeue

   Expected: Cluster and queue state are visible and stable.

2. Review role bindings and quota settings.

   kubectl get rolebinding,clusterrolebinding,resourcequota -A

   Expected: Bindings and quotas match approved governance model.

3. Run representative workload and validate policy behavior.

   kubectl apply -f priority-workload.yaml && kubectl get pods -A

   Expected: Workload scheduling follows expected priority and quota constraints.

Walkthrough: NGC registry administration check

Validate API key context and registry pull behavior from admin-runbook perspective.

1. Display current NGC configuration.

   ngc config current

   Expected: Expected organization/team and key context are active.

2. Test artifact pull path.

   ngc registry model list nvidia

   Expected: Registry query succeeds without auth error.

3. Rotate key and retest.

   ngc config set && ngc registry model list nvidia

   Expected: Post-rotation pull path remains functional.

Lab A: Scheduler admin health audit

Perform end-to-end scheduler administration checks and capture drift findings.

• - Validate control-plane and node health.
• - Review queue or scheduling policy health indicators.
• - Document one preventive action for detected risk.

kubectl get nodes -o wide

Nodes report Ready and expected software/runtime profile.

Lab B: RBAC and quota governance drill

Apply role and quota policy changes and verify enforcement behavior.

• - Create/modify role bindings and resource quotas.
• - Run allowed/denied operation tests.
• - Capture audit output and approvals.

squeue

Queue status reflects expected workload distribution and priorities.

Lab C: Model/data admin lifecycle exercise

Validate artifact version control and rollback readiness.

• - Register new model version and metadata.
• - Run integrity and access validation.
• - Simulate rollback to previous approved version.

cmsh -c 'show version'

BCM reports expected installed version and CLI access.

Lab D: NGC registry operations

Operate registry access securely with key rotation and validation.

• - Validate API key context before and after rotation.
• - Confirm artifact pull path from target nodes.
• - Verify no unauthorized access surfaces.

kubectl get rolebinding,clusterrolebinding -A

Bindings align with approved least-privilege model.