The GPU Training Failures That Hide From Your Dashboards

There is a prevailing assumption that machine learning training operates as a predictable batch process: engineers provision the hardware, initialize the cluster, and simply wait for the loss curve to converge over several weeks.

Operating at scale presents a fundamentally different reality. During the training of Meta's Llama 3 on a cluster of 16,384 H100 GPUs, the infrastructure experienced 419 unexpected interruptions over a 54-day period, equating to a failure roughly every three hours. While explicit hardware faults—such as degraded memory modules or failed boards—accounted for half of these incidents, the remaining disruptions were systemic and silent. These are the states where the application continues to run, but computational progress ceases, requiring engineers to manually correlate disparate telemetry to identify the bottleneck.

These implicit failures represent a massive financial sink. In a recent analysis, ByteDance reported logging nearly 6,000 jobs over a three-month window that continued to hold compute allocations and incur billing long after a fatal, yet silent, disruption occurred. On a standard 16-GPU cluster, spending a quarter of total operational time on stalled recovery or unproductive compute cycles translates to approximately $150,000 in wasted annual expenditure.

Addressing this requires a fundamental shift in how infrastructure teams approach observability, beginning with the deprecation of the industry's most relied-upon metric.

GPU Utilization Misdirection

The foundational metric exported by nvidia-smi, GPU-Util, is frequently misinterpreted as a measure of computational work. In reality, it strictly reports the percentage of the sampling window during which at least one kernel—the specific code executed by the application—was executing on the device. A single small kernel occupying one SM out of the device's many will report 100%. An application can peg this utilization metric at 100% simply by executing a continuous memory read/write loop, even if the actual arithmetic logic units remain entirely unutilized. Occupancy does not equate to progress.

The Hardware Reality: What is an SM?

Before dissecting the metrics that actually matter, it is necessary to define the hardware responsible for the math. The computational engine of an NVIDIA GPU is the Streaming Multiprocessor (SM). If the GPU is viewed as a sprawling manufacturing plant, the SMs are the individual assembly lines. A flagship data center card like the Hopper H100 contains up to 132 of these independent units.

Each SM is a highly complex core containing its own dedicated resources: standard CUDA cores for scalar arithmetic, Tensor Cores specifically designed for the dense matrix multiplications inherent to neural networks, L1 cache, and a massive register file. When PyTorch or JAX dispatches a workload, the kernel is broken down into thread blocks that are scheduled onto these SMs. True computational progress—the actual training of the model—only happens when these specific units are actively computing, rather than just waiting for instructions or memory transfers.

To accurately distinguish a GPU that is merely holding an allocation from one that is actively training a model, operators must monitor two specific fields from NVIDIA's Data Center GPU Manager (DCGM):

1. Graphics Engine Activity (DCGM_FI_PROF_GR_ENGINE_ACTIVE): Indicates if the broader compute engine is awake and issuing instructions. While standard GPU-Util merely measures if a kernel is resident, this metric tracks if the engine is actively processing. However, this still does not guarantee meaningful mathematical progress. For example, a script running a continuous memory bandwidth test, or a GPU busy-waiting during a stalled network transfer, will drive this metric to 100% because load/store instructions are constantly firing—even while the actual arithmetic units do zero work.
2. SM Activity (DCGM_FI_PROF_SM_ACTIVE): Measures the fraction of cycles in which at least one warp is resident on a Streaming Multiprocessor, averaged across every SM on the device. In the context of deep learning, this is the cheapest available proxy for whether work is genuinely spread across the machine. A high sustained value here (e.g., 80–90%) indicates the CUDA and Tensor Cores are occupied across a forward or backward pass, while a low value proves the hardware is starved for data or stalled at a barrier, regardless of what the broader engine metrics report. Note that it measures residency rather than arithmetic: to confirm that matrices are actually multiplying, pair it with the pipe-specific counters such as DCGM_FI_PROF_PIPE_TENSOR_ACTIVE.

When an allocation hangs, waits on an IO bottleneck, or spins inside a stalled all-reduce operation, the graphics engine activity will often read near 100%, but SM activity will drop to near 0%. The hardware is busy-waiting, not calculating. Without visibility into this distinction, infrastructure operators are entirely unequipped to diagnose silent stalls.

However, metrics alone are just numbers until they are correlated with application behavior. When we combine SM activity with kernel launch rates, collective communication logs, and host-level telemetry, the ambiguous failures that characterize distributed training stop being mysteries. They resolve into distinct, repeating patterns.

If you manage GPU infrastructure, these are the most expensive failure profiles you will encounter, how they masquerade as healthy runs, and the specific telemetry required to catch them.

1. The Checkpoint Stall

Because distributed runs fail so frequently, saving the model's weights and optimizer states to durable storage is a mandatory survival mechanism. The volume involved surprises people who assume this is a frontier-scale problem. Even a 32-billion parameter model, trained in mixed precision with FP32 master weights and Adam's two moment buffers, carries twelve bytes of state per parameter—roughly 384 GB written across the network on every save.

When checkpoints are not configured perfectly, they introduce a severe operational pathology that masquerades as a fatal failure. A synchronous checkpoint pauses every GPU in the cluster while the write operation completes. Against a storage backend delivering a modest 2 GB/s of aggregate throughput, those 384 GB take more than three minutes with every accelerator sitting idle. If the save interval is configured too aggressively relative to that throughput, the pauses compound into a serious bleed of compute efficiency. A three-minute pause every thirty minutes costs 2.4 hours of every day—which on a sixteen-node cluster of 128 GPUs is over 300 wasted GPU-hours daily, a full tenth of everything those nodes can produce. The cluster effectively turns into an extremely expensive data transfer pipeline.

During this stalling window, the compute telemetry goes cold—SM activity hits zero and kernel launches cease—but the cluster does not fall uniformly silent. While tensors are still being copied off the device, PCIe transmit traffic and memory-copy utilization run hot even though no arithmetic is being performed. Only once the data has left for storage does the card go truly idle, with power draw and temperature decaying toward baseline. Ranks that finish writing their shard early advance to the next collective and busy-wait there, pinning their engine activity near 100% against near-zero SM activity. Framebuffer usage, meanwhile, never moves at all: the allocation still holds tens of gigabytes on every card throughout, which is precisely why the scheduler and every memory-based dashboard continue to report a perfectly healthy job.

The result across a large cluster is not silence but incoherence—a fleet view in which some ranks appear dead while others appear saturated. To an automated cluster watchdog, or to an operator scanning the dashboard, this is easily misread as a fatal failure. If the run is cancelled on that assumption, hours of valid compute are needlessly destroyed.

The Diagnostic Key: Catching a checkpoint stall means correlating the cold GPU telemetry with the storage path running underneath it. Several independent signals fire during a save, and each one narrows the diagnosis further:

• Host-level file write bytes. Checkpoints are almost always written to network filesystems (like NFS, Lustre, or cloud object storage). By attaching an eBPF probe to the kernel's vfs_write path, or by monitoring network-filesystem client metrics, operators can observe the defining signature: cold GPU telemetry paired with gigabytes per second of sustained write throughput.
• PCIe transmit bytes (DCGM_FI_PROF_PCIE_TX_BYTES). This counter is measured from the GPU's perspective, so device-to-host copies register as transmit traffic. Before a single byte reaches storage, the state dictionary has to cross the bus—making a PCIe TX spike against flat SM activity the earliest marker of a checkpoint in progress, available without any host-side instrumentation.
• Kernel launch rate versus memcpy traffic. An eBPF probe on the CUDA API separates the various zero-progress states from one another. During a checkpoint, kernel launches fall to zero while device-to-host memcpy and stream-synchronization calls dominate the trace.

Measuring the duration of these bursts against the configured save interval then quantifies exactly how much of the run is being spent moving weights to storage rather than computing steps.

Best Practices: Modern infrastructure requires deprecating synchronous checkpoints entirely. Teams must implement asynchronous, staged checkpointing: the framework copies the tensors into host RAM, immediately releases the GPUs to resume computing the next step, and flushes the data to the network storage in a background thread. Furthermore, the checkpoint interval must be mathematically tuned against the cluster's Mean Time To Failure (MTTF)—saving just often enough to minimize lost compute without saturating the network.

2. The Distributed Hang

A synchronous data-parallel job is only as alive as its least responsive worker process (i.e. a rank). Every step funnels through a synchronization point where all GPUs in a process group exchange gradients (i.e. a collective), so the moment one participant stops making progress, every other GPU in that group stops with it—and because the groups are interlocked, the job as a whole. Critically, this is not registered as an error. Blocking inside a collective is a legal state, and from the library's perspective it is indistinguishable from waiting on a rank that is merely slow.

This is what makes hangs the most expensive failure class in the taxonomy: nothing exits. There is no exception to catch, because from the runtime's perspective nothing has gone wrong—a rank waiting on its peers is behaving exactly as designed, however long the wait lasts. The allocation remains healthy in the eyes of the orchestrator while every GPU in the allocation continues billing for a step that will never complete—frequently until a human notices that the loss curve stopped updating several hours earlier. What an operator actually sees is a job that has stopped without stopping: the GPUs are still retained by the scheduler, system logs are static, step counters are frozen, no traceback appears anywhere, and the high-level utilization meters may still be reporting 100% activity.

With a checkpoint stall ruled out—write throughput is zero—hangs manifest in one of three architectures, defined by the scope of the affected nodes:

• Rendezvous Hang: The entire process group fails to initialize simultaneously, typically due to network configuration issues such as a firewalled port or an unroutable coordinator IP address.
• Idle Hang: A single GPU stalls waiting on host-side operations, such as a blocked network filesystem read or a deadlocked data-loader thread.
• Frozen Collective: A single rank silently terminates (frequently due to an OOM-killed data worker). The surviving GPUs proceed to the next ncclAllReduce synchronization barrier and block indefinitely. In this state, their engine activity registers near 100% while SM activity collapses to a few percent—only the handful of SMs occupied by the spinning NCCL kernel.

The Diagnostic Key: The question is not whether the job is stuck, but which rank is stuck and on what, across a process group that may span a hundred ranks or more. Several signals answer it:

• The flight recorder. PyTorch maintains a per-rank circular buffer of collective records covering start events, completions, tensor sizes, and sequence numbers, which is written to disk when the watchdog fires. Collection is enabled through TORCH_NCCL_TRACE_BUFFER_SIZE, with the documentation recommending a buffer of 2000 entries, alongside TORCH_NCCL_DUMP_ON_TIMEOUT. The bundled analyzer then reads the per-rank dumps and enumerates the collective mismatches, naming the culprit rank and the operation on which it diverged. For the Frozen Collective in particular, this is the highest-value tool available.
• Collective sequence numbers. Every rank in a healthy process group works through the same operation count, so the fastest way to localize a hang is to find the rank whose count diverges. PyTorch fingerprints each collective with a sequence number, operation type, tensor shape, and dtype; running under TORCH_DISTRIBUTED_DEBUG=DETAIL surfaces mismatches directly, naming both ranks and the specific aspects on which they differ. Setting NCCL_DEBUG=INFO alongside NCCL_DEBUG_SUBSYS=COLL prints the collective call log itself, which is what catches hangs caused by mismatched operation types or message sizes. These modes carry a real cost—detailed debug and desync detection have historically introduced performance regressions at large scale—so they belong on a reproduction rather than across a production run.
• Host stack sampling. A sampling profiler attaches to a running process without stopping or restarting it, which is essential here because a deeply blocked process will ignore signals entirely. The practical pattern is to fan a py-spy dump out across every worker PID on the node at once and read the top frame of each: blocked in a filesystem read indicates an Idle Hang; blocked on a lock or futex indicates a deadlocked data loader; blocked inside an NCCL wait indicates a Frozen Collective. Because the informative frames frequently sit below the Python layer, a native-stack tool such as pystack, or strace at the syscall level, is worth keeping alongside it. Launching under torchrun with output teed and prefixed by role makes every line identifiable by node and rank, without which the merged output of a hundred-plus processes is unreadable.
• Kernel launch rate against engine activity. Launches at zero while engine activity holds near 100% is busy-waiting inside a collective. Both readings at zero indicate the process is blocked on the host and never reached the GPU at all. Collecting the launch side requires no framework changes: an eBPF uprobe on the CUDA runtime intercepts every submission from user space. Production agents extend the same pattern to NCCL entry points such as ncclAllReduce, aggregate the counts in a kernel-side hash map, and drain it every few seconds, which reduces data volume by one to two orders of magnitude against streaming each event individually. Pair the result with DCGM_FI_PROF_GR_ENGINE_ACTIVE for the engine side. The known limitation is that a CPU-side probe proves only that work was submitted, not that it executed—which is precisely the question at issue here.

Best Practices: The defining property of this failure class is that no component of the stack will report it, which makes continuous monitoring of the run itself the only practice that matters here. That monitoring has to be built on progress rather than liveness: a step counter that has not advanced in some multiple of the median step time is the sole honest signal available, because process existence, allocation state, and GPU-Util all continue to report a healthy job throughout every hang described above. What the alert triggers should be automatic rather than human—collective timeouts set tightly enough that the job terminates itself rather than idling for hours, allowing elastic restart to resume from the last checkpoint. That threshold does have to clear the longest legitimate stall in the run, which under synchronous checkpointing is the multi-minute pause described in section 1—one more reason to move that work off the critical path, and rank death propagated across the process group so that surviving GPUs never reach the barrier that would freeze them. Crucially, the diagnostic evidence must be captured before that teardown completes: flight recorder output written to node-local disk is purged the moment the allocation is reclaimed, taking with it the only record of which rank stopped first.

3. The Straggler and the PCIe Downgrade

Data-parallel training operates in strict lockstep, synchronizing gradients at a collective barrier during every step, which means the job as a whole advances at the speed of its slowest participant. If a cluster contains seven GPUs completing a step in 100ms and one GPU requiring 178ms, the seven healthy accelerators are forced into 78ms of idle wait time—on every step, for the entire duration of the run. Nothing about this registers as a fault. The job executes to completion without errors, simply at a significantly reduced throughput compared to identical historical runs on the same hardware tier.

This is the failure class now commonly termed fail-slow: a component remains fully operational but degrades enough to impair the job, without ever emitting an error code. It is not a rare event. One production study of clusters exceeding 10,000 GPUs found that well over half of jobs in the 512–1,024 GPU range encountered fail-slow stragglers, suffering an average completion delay in excess of a third of total runtime. The mechanism itself, though, has nothing to do with scale: the eight-GPU case above surrenders exactly the same fraction of every step. The reason these degraded nodes survive review is that conventional health checks—NCCL tests, GPU burn-in—verify functional correctness rather than sustained speed, so a slow accelerator passes qualification and returns to the production pool.

The Diagnostic Key: A straggler cannot be identified in isolation, because the affected rank is operating well inside every absolute limit its hardware defines. Every useful signal here is comparative:

• Per-rank step time, compared within parallelism groups. The core measurement is the dispersion of iteration times across ranks, quantified with a coefficient of variation to establish how inconsistent a group is and z-scores to name the outlier within it. Collect it with CUDA events rather than host-side timers: kernels launch asynchronously, so a wall-clock timer on the Python side measures submission rather than execution, and event-based timing at step granularity carries negligible overhead. The critical design constraint is that modern jobs run several parallelism dimensions simultaneously, so each rank must be compared only against ranks sharing the same parallel role—a tensor-parallel peer and a pipeline stage are not comparable quantities. Production systems pair two detectors over this series: a sliding window for short-term jitter and spikes, and change-point detection for step-wise regressions that would otherwise quietly become the new baseline. Note the blind spot in all outlier-based approaches: they assume only a few anomalous ranks per communication group, and lose statistical power entirely when the majority degrade at once, which is the case that requires a temporal baseline instead.
• Dispersion in per-rank SM activity. DCGM_FI_PROF_SM_ACTIVE, exported per GPU and grouped by rank, is the supporting signal. Because step counts and broad engine metrics are identical across all ranks, the straggler stands out only in the spread. Counterintuitively, the bottleneck GPU often displays the highest sustained SM activity, as it continues to process arithmetic while its peers wait idly at the synchronization barrier.
• Relative scoring against historical scoring. Purpose-built detectors compute two distinct scores: each rank's performance relative to the other ranks in the same run, and each rank's performance against its own history over time. The first catches a single bad accelerator in an otherwise healthy fleet; the second catches gradual or uniform degradation that a purely relative comparison would normalize away. NVIDIA ships this as a straggler detection API in its resiliency extension, which wraps the training step function, instruments it to measure CUDA kernel execution times, reports the affected ranks by index, and can be configured to checkpoint and terminate the run outright.
• PCIe link width and generation, current against maximum. While stragglers are often blamed on slow host-side data loaders, they are frequently the result of a degraded PCIe link that has silently downtrained from x16 to x8 or x4, severely restricting host-to-device bandwidth. DCGM exposes the pair as separate fields for exactly this comparison—DCGM_FI_DEV_PCIE_LINK_WIDTH against DCGM_FI_DEV_PCIE_MAX_LINK_WIDTH, and DCGM_FI_DEV_PCIE_LINK_GEN against DCGM_FI_DEV_PCIE_MAX_LINK_GEN—so the alert is simply current below maximum. Evaluate it under sustained load, however: links legitimately downtrain at idle for power management, so a mismatch on a quiescent GPU should be corroborated against throughput and replay counters before anyone is paged. The same values are available from nvidia-smi -q -d PCIE, from lspci -vv as link capability versus link status, and from the kernel's sysfs entries for maximum and current link width. NVIDIA's diagnostics treat insufficient width and generation as their own named failure conditions rather than as performance observations.
• PCIe replay counters and correctable error counts. DCGM_FI_DEV_PCIE_REPLAY_COUNTER records retries caused by transmission errors on the bus, with DCGM_FI_DEV_PCIE_COUNT_CORRECTABLE_ERRORS alongside it. Both are cumulative counters, so apply a rate function rather than alerting on the absolute value; on healthy hardware the rate should be flat at zero, and a climbing rate indicates an unreliable physical link rather than a software bottleneck. NVIDIA's own stress diagnostics fail a GPU when replays exceed a defined ceiling over the test window, which makes a reasonable starting threshold. Pairing this with DCGM_FI_PROF_PCIE_RX_BYTES confirms the consequence directly: a downtrained link caps host-to-device throughput well below the tier's expected ceiling.

Best Practices: Because a fail-slow rank never crosses an absolute limit, detection has to be continuous and comparative rather than threshold-based—per-rank step times evaluated against peers in the same parallelism group and against the job's own history, collected for the life of the run rather than sampled after someone notices the throughput regression. That monitoring should terminate in an action rather than a dashboard: checkpointing and evicting the offending node, since a degraded rank left in place taxes every other GPU in the job for as long as it remains. The same measurement also belongs on the other side of the job boundary. Since functional health checks pass degraded hardware, nodes should be qualified on measured performance before entering the production pool—an offline sweep that has been reported to cut run-to-run step time variance from roughly 20% to 1% while raising mean FLOPs utilization by as much as 1.7x.

4. The Communication-Bound Run

A data-parallel step ends by exchanging gradients, which places the collective squarely on the critical path: while it is in flight, the SMs have nothing to compute. When that exchange slows, the arithmetic units spend the majority of each step waiting for synchronized gradients rather than producing updates. Because the collectives do eventually complete, nothing about this registers as a failure. Overall throughput drops to a fraction of expected performance, GPU dashboards continue to show high activity, no exceptions are raised, and the loss curve descends exactly as it should—only more slowly, at full hourly cost.

Communication is also where degradation is most likely to occur and most likely to persist. Production benchmarking has found slow-communication faults to be markedly more common than slow-computation ones and roughly twice as long-lived, typically originating in congestion or a marginal link rather than in the accelerator itself. Two mechanisms account for most of it:

• Silent Fallbacks: An improperly configured environment variable (such as an errant NCCL_IB_DISABLE) or an undetected network adapter failure can trigger NCCL to silently abandon InfiniBand and fall back to standard TCP over Ethernet.
• Hardware Degradation: A compromised NVLink connection experiencing high error rates and constant recovery cycles will artificially gate the speed of the entire communication ring.

The Diagnostic Key: Every signal that matters here lives on the network rather than on the accelerator, which is precisely why GPU-centric observability platforms report a healthy job throughout:

• The transport NCCL actually negotiated. Running with NCCL_DEBUG=INFO forces the library to print its chosen startup topology, explicitly detailing the transport layer in use; NCCL_DEBUG_SUBSYS=INIT,NET,GRAPH narrows the output to the relevant subsystems. The single most valuable line identifies whether a connection came up on InfiniBand or downgraded to socket transport. Because this is emitted once at initialization, it should be parsed and asserted automatically at job start rather than read by a human after a throughput regression has already cost a week of compute.
• NVLink error counters, evaluated as rates. DCGM exposes four distinct counters—DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR_COUNT_TOTAL for flow-control CRC errors, DCGM_FI_DEV_NVLINK_CRC_DATA_ERROR_COUNT_TOTAL for data CRC errors, DCGM_FI_DEV_NVLINK_REPLAY_ERROR_COUNT_TOTAL for retries, and DCGM_FI_DEV_NVLINK_RECOVERY_ERROR_COUNT_TOTAL for recovery events. Per-link variants exist for each, which localize the fault to a specific lane rather than a whole GPU, and dcgmi nvlink --errors prints the same breakdown interactively. All four are cumulative, so alert on the rate: NVIDIA's own health checks treat sustained CRC errors above roughly a hundred per second as a distinct condition, and usefully separate an unacceptable error rate—where the link still carries workload—from a link that is down outright. Be aware that these field names were shortened in recent DCGM releases, so verify them against the version actually deployed.
• NVLink throughput against the link's ceiling. DCGM_FI_PROF_NVLINK_TX_BYTES and DCGM_FI_PROF_NVLINK_RX_BYTES give aggregate traffic, and per-link fields summed as TX plus RX give the bandwidth of an individual link. A ring gated by one degraded connection shows depressed aggregate throughput long before any counter crosses a failure threshold.
• InfiniBand port counters. These are read from the kernel at /sys/class/infiniband//ports//counters/ and are collected by the standard node exporter's InfiniBand collector, so no bespoke agent is required. Track port_xmit_data and port_rcv_data for throughput—noting that the raw sysfs values are counted in four-byte words rather than bytes, a conversion the node exporter applies for you and a hand-rolled scraper will not—and link_downed, link_error_recovery, local_link_integrity_errors and symbol_error for health. Symbol errors count minor link errors on individual physical lanes, which makes them the earliest available warning that a cable or transceiver is becoming marginal—well before the link training state machine gives up and downs the port.
• The duration of the collective itself. Because the operations complete, the job does not halt; instead, standard CUDA-API call latency expands from typical microsecond ranges into hundreds of milliseconds. Collect this without touching framework code using eBPF uprobes on the NCCL entry points, which yields per-collective flight times directly. PyTorch can also record durations natively by enabling collective timing, which inserts CUDA events at the start of each operation—though at that density, hundreds of events per step rather than the handful used for step timing, it carries real CPU overhead and, because it issues event queries, can itself increase the likelihood of a watchdog hang.

Best Practices: The controlling principle is that the fabric must be verified rather than assumed. Transport selection should be asserted programmatically at startup—parse the initialization output and fail the job immediately if it did not come up on the expected network, since a run that silently downgrades to TCP will otherwise complete normally and simply bill several times over. From there, collective duration belongs in the monitoring stack as a first-class metric sitting alongside step time, with link error counters alerted on their rate rather than their absolute value, because every counter discussed here is cumulative and a slow, steady climb is the signature that matters. Finally, achieved bus bandwidth should be periodically compared against a known-good benchmark for the same fabric and message size; without that baseline there is no way to distinguish a communication-bound run from a job that was simply always this slow.

5. Quietly Slow Silicon

A GPU that reaches one of its operating limits does not fail; it protects itself by aggressively downclocking, and continues reporting perfect health while doing so. Training loss curves appear normal, high-level utilization is maximized, and system logs indicate healthy execution, but the absolute step throughput is substantially degraded. Data center GPUs expose several distinct temperature thresholds and they are easy to conflate: the target temperature, often around 83°C, is only the setpoint the cooling loop aims at, while hardware slowdown engages at the card's higher slowdown threshold—typically in the high 80s to low 90s depending on the SKU. The distinction matters because hardware slowdown is not a gentle taper. NVIDIA documents it as reducing core clocks by a factor of two or more, and it does so without ever logging an application-level warning.

Temperature, however, is only one of the triggers, and treating throttling as a synonym for overheating hides an entire class of causes. A GPU running at a comfortable 55°C can be clocked down just as hard by a power constraint, and the cause chain is completely different. The enforced power limit may simply have been set below the board's rated draw—by an administrator, a cloud provider, or a rack-level power budget—at which point the driver's software power-scaling algorithm reduces clocks to stay underneath it. Alternatively, the system power supply can assert an external power brake, halving core clocks in response to a facility-level constraint that has nothing to do with the accelerator at all. There is also a coupling effect worth knowing about: GPUs placed in a sync boost group all drop to the lowest clock achievable across the group, so a single degraded card silently holds down every one of its neighbors, whose own telemetry will look blameless.

The Diagnostic Key: Standard utilization metrics remain pinned at 100% throughout, so the entire diagnosis rests on frequency and on the reason behind it:

• Clock frequency against the rated ceiling. DCGM_FI_DEV_SM_CLOCK compared against the board's specified boost clock is the primary effect measurement, with DCGM_FI_DEV_MEM_CLOCK alongside it. This tells you the GPU is slow but not why, which is why it is never sufficient on its own.
• The throttle reason bitmask, decoded rather than merely alerted on. DCGM_FI_DEV_CLOCK_THROTTLE_REASONS is a bitmap mapping one-to-one onto the driver's own reason codes, and the individual bits point at completely different remediations. Software power cap indicates the power-scaling algorithm is holding clocks below what was requested. Hardware and software thermal slowdown indicate temperature, the latter firing when the GPU exceeds its maximum operating temperature or when memory breaches its own threshold. Hardware power brake indicates an external assertion from the system power supply. Sync boost indicates this GPU is fine and is being held back by another in its group. One bit is deliberately ambiguous: generic hardware slowdown covers excessive temperature, an external power brake, and fast-trigger protection against high power draw all at once, so it should be treated as a prompt to look at temperature and power together rather than as a diagnosis. Because the field is a bitmap, values combine additively, and a dashboard that renders it as a raw integer will show sums rather than causes.
• Power draw against the enforced limit. DCGM_FI_DEV_POWER_USAGE pinned flat against DCGM_FI_DEV_ENFORCED_POWER_LIMIT is the signature of a power-capped card, and comparing the enforced limit against the board's default is what reveals a cap that was set too conservatively in the first place. NVIDIA's diagnostics carry a dedicated failure condition for exactly this—an enforced power limit too low to reach the target performance—which is worth mirroring as an alert, because nothing else in the stack will mention it.
• Cumulative violation-time counters, converted to rates. DCGM_FI_DEV_POWER_VIOLATION and DCGM_FI_DEV_THERMAL_VIOLATION record accumulated time spent throttled for each reason, with companion counters for board limit, reliability, sync boost, and application and base clock limits. Applying a rate function over these cumulative counters yields the true percentage of time the hardware spends in each throttled state, preventing short throttling spikes from being hidden by slow metric polling intervals. Check the units against the deployed DCGM version, as the violation fields have been documented in both microseconds and nanoseconds across releases.
• Temperatures, memory included. DCGM_FI_DEV_GPU_TEMP is the obvious one, but DCGM_FI_DEV_MEMORY_TEMP matters independently: HBM can breach its own limit while the core temperature still reads comfortably, and software thermal slowdown fires on either. Both should be evaluated against the card's published slowdown threshold rather than against a hand-picked number.

Best Practices: Because every reason produces the same symptom, the practice that matters is alerting on the decoded cause rather than on the slowdown—a thermal bit sends someone to check airflow and cold plates, a power cap bit sends someone to check a configuration value, and a power brake bit is a facility problem that no amount of GPU work will fix. The enforced power limit in particular should be asserted at job start against what the hardware tier is supposed to deliver, in the same way the network transport is verified, since a conservative cap applied cluster-wide is invisible, permanent, and silently taxes every run. Beyond that, achieved clock frequency is worth tracking as a fleet-wide baseline rather than a per-node threshold: a single node sitting consistently below its peers identifies a hot aisle, a failing cold plate, or a bad power delivery path long before any counter crosses a limit. And when the sync boost bit appears, the investigation belongs to the group rather than the GPU that reported it.

Putting the Pieces Together

Managing ML infrastructure at scale requires discarding the assumption that exit codes and utilization bars provide a complete picture of cluster health. Diagnosing these expensive compute sinks requires correlating telemetry across multiple, distinct layers of the stack:

1. eBPF Probes: To instrument kernel launch rates and collective flight times without modifying the user's framework code.
2. DCGM and Hardware Counters: To explicitly track SM activity, PCIe link degradation, and hardware throttling states.
3. Standard Output Streams: To parse network topology initializations (NCCL) and monitor the mathematical health of the loss curve.
4. Scheduler Telemetry (Slurm/K8s): To accurately differentiate true node hardware failures from routine scheduler preemptions.

Evaluating these signals concurrently—before the ephemeral node allocation is released and local logs are purged—is the modern baseline for ensuring training reliability. While specialized observability platforms are emerging to automate this exact correlation, the core engineering principle remains unchanged:

High occupancy does not guarantee productivity, and a rented GPU is not inherently a computing one.

What RidgeScope Does With This

Correlating these signals across every rank of a live run is what we build RidgeScope to do. Every failure mode above shares a structural property: none of them is visible in a single metric. A checkpoint stall looks like a dead job until write bytes are placed next to SM activity; a frozen collective looks like a saturated GPU until kernel launches are placed next to engine activity; a straggler looks like a healthy fleet until per-rank step times are compared within their parallelism group.

Our lightweight on-node agent deploys across the cluster—no SDK, no code changes, and no access to training code, datasets, or model weights—and collects the four layers this article has been describing.

AI investigations then run from the moment a job starts, so an alert arrives with the verdict already attached: which rank, which layer, and what specifically happened, with every claim cited to the metric or log line behind it. The failure surfaces in minutes rather than at the end of a run, before it has consumed another night of GPU hours.