20+ ways training wastes compute
Idle allocations, busy-waiting collectives, slow ranks, checkpoint storms, failed hardware, and runs that finish without producing a useful model. RidgeScope names the waste mechanism, shows the evidence, and explains the next action. Select any mode to see how it is judged.
GPU under-utilization (input-pipeline starvation)
Kernels launch, loss decreases — and the duty cycle is low, with idle gaps between bursts. The input pipeline cannot feed the card. The run usually completes cleanly, slower than it should, and nothing else flags it. Baseline-free: severe under-utilization is confirmed from the run alone.
The GPU is launching kernels and the loss is decreasing, but the engine duty cycle is low with recurring step-scale launch gaps — the GPU sits idle a large fraction of wall-time. The hardware is clean.
The run is under-utilizing the GPU, most likely input-pipeline starvation. Baseline-free, we can confirm a severe under-utilization (near-zero engine while launching, or multi-second hard stalls) but only suspect a mild one — a small or CPU-bound model can legitimately under-fill a large GPU.
Compare this run with a healthy run or throughput target. RidgeScope identifies the utilization gap immediately; the comparison distinguishes data starvation from the workload’s normal compute profile and quantifies recoverable capacity.
GPU under-utilization (input-pipeline starvation)
Kernels launch, loss decreases — and the duty cycle is low, with idle gaps between bursts. The input pipeline cannot feed the card. The run usually completes cleanly, slower than it should, and nothing else flags it. Baseline-free: severe under-utilization is confirmed from the run alone.
The GPU is launching kernels and the loss is decreasing, but the engine duty cycle is low with recurring step-scale launch gaps — the GPU sits idle a large fraction of wall-time. The hardware is clean.
The run is under-utilizing the GPU, most likely input-pipeline starvation. Baseline-free, we can confirm a severe under-utilization (near-zero engine while launching, or multi-second hard stalls) but only suspect a mild one — a small or CPU-bound model can legitimately under-fill a large GPU.
Compare this run with a healthy run or throughput target. RidgeScope identifies the utilization gap immediately; the comparison distinguishes data starvation from the workload’s normal compute profile and quantifies recoverable capacity.
