evaluation cases passed
Correct family and run status across the current H100 and Slurm regression suite.
Evidence behind productive GPU utilization27 17 0
A busy GPU can still be wasting money.
Most GPU dashboards tell you whether a device is active. RidgeScope shows whether every allocated GPU-hour is producing useful training.
It connects scheduler allocation, host resources, storage I/O, CUDA execution, distributed progress, and model learning—then cites the evidence behind every conclusion.
100% GPU activity can mean zero useful compute. Exit code 0 can mean a useless model.
failure families
From silent underutilization to distributed hangs, bad outcomes, and hardware faults.
training code changes
No SDK integration or training source-code access required.
Follow every allocated GPU-hour01 02 03 04 05
Where does productive utilization stop?
An allocation only creates value when infrastructure supplies the workload, kernels perform useful computation, distributed workers advance together, and the model actually learns.
Allocated
Were GPUs reserved?Slurm allocation, job lifecycle, nodes, GPUs, CPUs, memory, requeues, and exit status.
Supplied
Were the GPUs kept fed?Host CPU and RAM, throttling, storage I/O, pressure, disk delay, and file writes.
Computing
Was activity useful compute?SM activity, kernel launches, tensor pipelines, memory transfers, and launch gaps.
Advancing
Were all ranks progressing?NCCL collectives, rank balance, InfiniBand, NVLink, and recent kernel progress.
Learning
Did the run improve the model?Steps, loss, gradient norms, checkpoints, resume markers, and terminal outcome.
What ordinary dashboards miss
The utilization number is only the beginning.
Examples from RidgeScope's live H100 and Slurm evaluation suite.
4.9GPU-hours held
0 kernels launched
The allocation remained active for almost five hours, but no CUDA process attached.
Completely idle allocation
94%engine activity
<1% SM activity
The devices appeared busy while recent kernel and collective progress had stopped.
Spin-waiting, not training
15GPUs waiting
1 rank at 3.5× median
The run advanced, but one slow rank repeatedly stalled every peer in the allocation.
Straggler-bound training
0exit code
Loss finished 6.6× worse
The infrastructure completed the work successfully. The training outcome did not.
Completed, but not productive
See what is starving the GPUs01 02 03 04
The GPU is often where waste appears—not where it begins.
Low utilization can originate in the host, data pipeline, storage layer, scheduler, or Kubernetes environment. RidgeScope captures the surrounding evidence needed to separate them.
Checkpoint & storage I/O
Workload-attributed file writes show when lost compute time coincides with NFS, local, tmpfs, or FUSE activity—alongside disk throughput, delay, capacity, and I/O pressure.
CPU & data loading
CPU usage, limits, throttling, scheduling delay, launch gaps, and host-to-device transfers help separate slow input delivery from an inherently small workload.
Host memory
Node and container RSS, cache, limits, swap, NUMA, and memory pressure reveal when host RAM—not GPU memory—is constraining training.
Kubernetes constraints
Pod state, restarts, termination reasons, requests and limits, node conditions, scheduling, and persistent volumes explain control-plane constraints around the run.
From symptom to bottleneck
Know when the GPU is waiting on storage.
A GPU dashboard shows the utilization drop. Workload-attributed I/O connects the drop to checkpoint activity and the filesystem serving it.
- 1GPUs remain allocated
- 2SM activity collapses
- 3Workload file writes spike on NFS
- 4Checkpoint logs align with the stall
- 5The evidence points to storage-bound training
One verdict, fully evidenced
“The GPUs are busy” is not a diagnosis.
16 GPUs allocated15 affectedfinal 120 seconds
Collective hang after one rank terminated
The surviving ranks remained highly active, but almost none of that activity was useful model computation.
15 GPUs were spin-waiting, not training.What the data shows
- Sixteen GPUs remained allocated.
- One rank terminated unexpectedly.
- Survivors reported approximately 94% engine activity.
- SM activity fell below 1%.
- No kernels or NCCL collectives completed in the final 120 seconds.
What it rules out
- The run was not making slow forward progress.
- GPU memory was not exhausted.
- No coincident GPU-health or node-down event explained the stall.
- High engine activity did not represent useful compute.
How to confirm the cause
Correlate the termination timestamp with host memory pressure, cgroup events, and watchdog or audit logs to distinguish a resource kill from an external termination.
The signals behind every verdict
One timeline from allocation to outcome.
Scheduler & allocation
Slurm job, step, node, and controller- Lifecycle, queueing, pending reasons, requested and allocated resources, requeues, preemption, and exit status
- Node state, job placement, elapsed time, restart count, submit line, and I/O paths
What was reserved, where did it run, and for how long?
GPU & CUDA execution
physical GPU, process, and kernel- Engine and SM activity, occupancy, compute pipelines, memory, power, clocks, throttling, XID, and ECC
- CUDA errors, allocation failures, kernel cadence, launch gaps, API duration, and memory transfers
Was the GPU active—and was it performing useful computation?
Distributed training & fabric
rank, communicator, link, and node- NCCL collectives, communicator state, in-flight work, completion counts, and per-rank progress
- InfiniBand, PCIe, NVLink, retransmissions, congestion, link state, and fabric errors
Were workers advancing together, waiting on a rank, or blocked by communication?
Storage & data pipeline
workload, filesystem class, and device- Regular-file write volume and operations attributed to the workload and classified as NFS, local, tmpfs, FUSE, or other
- Cgroup reads and writes, disk delay, I/O pressure, device throughput, filesystem capacity, NFS, and NVMe
Was compute interrupted by checkpoints, storage, or input delivery?
CPU & memory
node, container, service, VM, and process- CPU usage, limits, throttling, scheduling delay, load, frequency, and pressure stall information
- RSS, cache, limits, swap, NUMA, page faults, memory pressure, and process behavior
Were GPUs starved by host resources?
Kubernetes state
cluster, node, workload, pod, and container- Pod status, restarts, termination reasons, requests and limits, ownership, and live CPU and memory usage
- Node conditions, capacity, scheduling, Jobs, workload controllers, persistent volumes, and storage classes
Did the orchestration environment constrain or interrupt the workload?
Training progress & logs
stdout, stderr, framework, and rank- Attributed exceptions, tracebacks, steps, throughput, loss, gradient norms, learning rate, and overflows
- Checkpoint saves, evaluation output, early stopping, resume markers, and distributed launcher summaries
Was the run merely executing—or was the model actually learning?
Topology & attribution
the map connecting every signal- GPU, process, rank, container, pod, Kubernetes workload, Slurm job, step, task, node, filesystem, and fabric
Which workload caused the behavior, and which resources paid for it?
Evidence you can trust01 02 03 04
RidgeScope does not guess.
RidgeScope does not guess when evidence is incomplete. It shows what is proven, what is ruled out, and the next evidence needed to confirm the cause. Every conclusion must survive comparison across scheduler state, infrastructure, GPU execution, distributed progress, and training outcome.
What the data shows
Exact measurements, affected resources, time windows, and supporting queries.
What it implies
The explanation best supported by the combined evidence.
What it rules out
Counterfactual evidence showing why competing explanations do not fit.
How to confirm the cause
The next measurement or operator check that resolves the remaining ambiguity.
