Why CUPTI and NCCL Monitoring Is Hard

The most useful question in distributed training is often the hardest one to measure: what was each GPU actually doing while the job appeared to be running?

Node-level telemetry can tell us that a GPU was allocated, powered, warm, and executing something. It cannot tell us whether an all-reduce was making progress, whether a rank spent the interval spinning inside a communication kernel, or which kernels consumed the device timeline. CUPTI and NCCL can expose that missing layer. They can also change the timing of the process they observe, collide with tools already inside it, disappear behind a version mismatch, or—in the worst case—help wedge the workload.

This is why production GPU monitoring cannot be designed as “turn on every profiler and export the result.” The correct design is a ladder: begin with signals that are safe everywhere, add semantic detail only when the runtime supports it, and keep every invasive tier optional, observable, and able to fail without taking the training run with it.

This article explains the engineering constraints behind that ladder, the failure modes we encountered on real multi-GPU training, and the architecture we use in RidgeScope. It deliberately stays above implementation details. The goal is to make the trade-offs clear enough for anyone building a cluster-wide collector to choose an honest boundary.

The first trap: host time is not device time

CUDA is asynchronous by design. A host thread usually submits work to a stream and continues before the GPU has completed it. NCCL follows the same model: a collective call returns after the operation has been enqueued to its CUDA stream, while the communication kernels execute later on the device.

That gives one operation several valid durations:

• Host API duration measures how long the CPU spent entering the runtime, preparing work, waiting for enqueue dependencies, and returning.
• Queue delay measures how long submitted work waited behind earlier stream work before it could begin.
• Device execution time measures the interval in which the kernel was active on the GPU.
• End-to-end collective time may include rank arrival skew, network progress, proxy work, kernel execution, and final synchronization.

None is a substitute for the others. Host duration is excellent for detecting a call that blocks unexpectedly, communicator initialization that never returns, or a rank that stops issuing collectives. It is not a reliable denominator for achieved bus bandwidth because the bytes can still be moving after the host call returns. Device kernel time is the better denominator for GPU-side communication, but it includes time spent waiting inside a persistent or spinning NCCL kernel. That is honest device occupancy, not necessarily useful byte movement.

The word attributed matters. NCCL knows the collective type, element count, datatype, communicator, rank, algorithm, and transport decisions. CUPTI knows when a kernel ran and on which device or context. A useful monitor must join semantic information from one layer to timing from another without accidentally joining unrelated work from adjacent streams or CUDA graphs.

The safest timestamp is not always the timestamp that answers the question. The most precise timestamp is not always safe enough to collect continuously.

There is no single NCCL monitoring API

The phrase “NCCL metrics” hides several fundamentally different sources. Each sees a different part of the operation and has a different failure boundary.

Source	What it establishes	What it cannot establish alone
DCGM and NVML	The device was busy, memory-resident, throttled, unhealthy, or moving link traffic	Which job operation caused the state, or whether a collective progressed
Host CUDA/NCCL probes	A process called an API, how often, how long the host stayed inside it, and whether it returned	When asynchronous device work began or ended
Passive NCCL state (research)	Communicators still exist; progress counters advance or freeze; one rank diverges from peers	Portable per-collective timing across unknown binaries
NCCL profiler plugin	Collective identity, message size, communicator context, and version-dependent event timing	Coverage on older NCCL builds or jobs already initialized without it
CUPTI Activity	Device-side kernel and memory-operation records with GPU timestamps	Collective meaning unless records are correlated or classified
CUPTI counter profilers	Detailed hardware behavior inside selected ranges or kernels	Low-risk coexistence with continuous DCGM counter collection

RidgeScope's shipped late-attach hang path combines host-side NCCL collective and communicator-initialization state, CUDA launch gaps, and independent engine and SM behavior across ranks. It works without restarting the job. The opt-in device tier adds exact kernel timing and classification; it is not the first point at which RidgeScope can detect a distributed wait.

A robust diagnosis uses several rows. Engine-high and SM-low telemetry says that a GPU is occupied without broad arithmetic work. A host probe showing an in-flight collective says where the process is waiting. Passive progress or a device-side NCCL kernel that stops advancing says the wait is not merely a slow enqueue. Scheduler identity tells us whether all of those signals belong to the same job and rank.

Attach timing is part of the data model

A fleet monitor is expected to discover work after it starts. Profiling systems are often designed around the opposite assumption: the tool launches the program and owns its lifecycle.

Host probes are comparatively forgiving. A privileged node agent can discover a loaded CUDA or NCCL library, attach to stable symbols, and begin observing subsequent calls. The first part of the run is missing, but the running process does not need to restart. This makes host probes useful for incident response and for jobs launched without special configuration.

CUPTI itself supports dynamic initialization when code inside the target process invokes its API. That does not make external late attach equivalent to process-start instrumentation. In one public experiment, code injected into a live process initially received host API activity but no concurrent-kernel or memory-copy activity; the author later reported getting the device records to work, but did not publish the corrective details. The result is evidence that this path is fragile and under-specified, not that it is impossible. For transparent, reproducible fleet coverage, process-start injection remains the reliable route: the monitoring component is present when CUDA initializes, before the activity of interest occurs. A job launched without it cannot later gain a complete device timeline through agent-side configuration alone.

NCCL profiler plugins have a similar lifecycle boundary. NCCL selects and loads the profiler as communicators initialize. A missing or failed plugin at that point is not something an out-of-process agent can repair after the job has built its communicator graph. The exact behavior varies by NCCL release, which makes the startup log and an explicit coverage signal essential.

The runtime inside the job is the version that matters

GPU nodes often contain several CUDA stacks at once: a host driver, a system toolkit, container libraries, and framework wheels that bring their own CUDA and NCCL dependencies. The library mapped into the training rank wins. Looking at the node's default toolkit can therefore produce a perfectly consistent—but irrelevant—compatibility report.

CUPTI makes this especially important. Activity record structures evolve. New structure versions replace older ones, fields move behind size gates, previously unsafe callback behavior is fixed, and APIs gain or lose compatibility with concurrent features. The CUPTI API version is distinct from the CUDA runtime version. A collector built against one set of headers cannot assume that every library sharing the same major soname has the same record layout or behavior.

We learned to treat every CUDA minor release as a separate compatibility target. The collector loads the CUPTI library belonging to the workload environment. Its compatibility control is a build-and-acceptance process, not runtime negotiation of unknown layouts: it is built against known activity records, uses the same conservative activity subset on accepted runtimes, and drops record kinds it was not built to handle. We validated that path on CUPTI 12.0.146 and 12.4. For another CUDA minor, RidgeScope runs one 25-minute, workload-shaped smoke job on the client's own stack as part of onboarding. This gate applies only to the opt-in device-timing tier; fleet telemetry and late-attach host probes carry no such per-minor gate. Shipping a private copy ahead of the workload's own CUPTI looked simpler, but it created a more dangerous state: the monitor could silently shadow the framework's expected library.

NCCL has the same shape. The profiler interface appeared and then expanded across releases. GPU-generated kernel timestamps arrived in the 2.27 line; richer API and kernel event support and the Inspector example followed in the 2.28 line. A cluster with older framework environments cannot gain those capabilities because the host has a newer NCCL package. Upgrading the monitor does not upgrade the library already linked into the job.

Build-time layout discipline is necessary but not sufficient. We also gate features by behavior. A release note can say that an activity kind exists; only a workload-shaped acceptance run shows whether it coexists with CUDA graphs, high launch rates, checkpoint pauses, multi-rank synchronization, and the profilers already used by the framework.

Coexistence is a resource-allocation problem

Observability tools compete for resources that are not virtualized cleanly.

The subscriber slot

Before CUDA 13.3—with multi-subscriber support available only as a beta in 13.2—production CUPTI activity tracing effectively allowed one subscriber. PyTorch's profiler, Kineto, Nsight Systems, and a cluster monitor could all want that same position. Startup order determined the winner. A monitor that treated “subscriber unavailable” as a fatal error could break the application; one that took the slot first could disable the developer's intended profile.

CUDA 13.3 promoted multi-subscriber tracing to production, but it is not a magic compatibility switch. All participants must use the newer subscriber-scoped API, the first subscriber establishes process-wide policy, and the feature applies to activity and callback tracing rather than the hardware-counter profiling APIs. Mixed old and new tools still need an explicit policy.

This is the primary reason RidgeScope keeps device timing client-controlled and opt-in. On a legacy single-subscriber runtime, process-start instrumentation can reserve the slot before a profiler launched later by the framework or user. Until coexistence policy is explicit for a deployment and tested against its profiler stack, the safe operational choices are to instrument selected jobs, avoid overlapping profiler sessions, and retain host probes and fleet telemetry as the fallback.

The hardware counter lock

DCGM profiling fields use GPU hardware counters to provide low-overhead interval averages for SM activity, Tensor Core activity, memory activity, PCIe, and NVLink. CUPTI's range profiler, PM sampling, and related counter APIs want overlapping hardware resources. NVIDIA documents that DCGM profiling can conflict with developer profiling tools and provides a host-wide pause/resume mechanism.

That mechanism is appropriate for a scheduled, focused profiling session. It is a poor default for unattended observability because pausing DCGM creates blind spots for every job on the host, not only the process being inspected. Counter profiling can also replay kernels or serialize work depending on the mode, changing the behavior under measurement.

We keep the counter tier out of continuous collection. Broad DCGM telemetry remains on; CUPTI Activity supplies timestamps without entering replay-based profiling; deeper counter work belongs to an explicitly scheduled diagnostic window.

The NCCL plugin position

NCCL also selects a profiler plugin rather than broadcasting events to an unlimited set of observers. A site-wide plugin can conflict with a profiler chosen by the workload. The plugin's own output path is part of the reliability design: unbounded JSON lines can fill shared storage, while a small in-memory ring can overwrite records during a burst. A community measurement submitted to NCCL's Inspector development found that its default 1,024-entry per-communicator ring captured only 48.4% of collectives at about 770 collectives per second on a two-node, 16-H100 test. Overwrites produced no loss counter or log. This was not a RidgeScope deployment result—our validated cluster runtime predates Inspector—but it is directly relevant to any always-on design.

An official plugin interface is not automatically a low-risk alternative. Public NCCL reports describe hangs and crashes in Inspector and example-plugin paths, including a proxy progress thread deadlocked inside an Inspector kernel-event callback and a failure in NCCL core with a do-nothing plugin. These reports do not invalidate the interface; they define the acceptance boundary. The recorded failures concentrate in proxy- and kernel-event tiers, while the application-thread event tier has no matching incident in the record we reviewed.

The bypass is not to make the ring infinitely large. Begin with the cheapest application-thread events, drain them out of process, measure overwrite and queue age, and enable proxy- or kernel-level events only after a workload-shaped soak. If another plugin is already selected, fall back to host probes and passive signals.

When the observer became the failure

The most important result in our CUPTI work was a feature we removed.

We wanted more than kernel start and end times. CUPTI exposes optional latency timestamps intended to separate when a kernel was queued, submitted, and executed. That looked like the cleanest way to quantify launch delay without application changes.

On our own eight-H100 acceptance cluster during the proof of concept, enabling that option caused a deterministic failure. We reproduced it twice across CUPTI 12.0 and 12.4. Activity-buffer completion stopped. The device-side tracing pool stopped recycling. A CUDA context eventually stopped retiring kernels, while peer ranks remained inside NCCL work waiting for it. The framework's NCCL watchdog killed the run ten minutes later. From the outside, the result looked exactly like an ordinary distributed hang.

The mechanism matches a CUPTI latency-timestamp hang reported to NVIDIA in June 2024: buffer requests continued after completion callbacks stopped, and the process stopped making progress. NVIDIA initially could not reproduce that report, and the public support thread remains unresolved. Our workload supplied a deterministic multi-GPU reproducer across two CUPTI versions, which let us isolate the unsafe option and design it out of the production path.

The option was not merely unsafe; it was not delivering dependable value. Separately, in a single-GPU stress test, the native submitted timestamp was populated for only about a quarter of the records. We banned the mode from the production collector instead of hiding it behind an experimental switch.

We then repeated the failure on a disposable acceptance job with the tracing buffers placed in pinned host memory. In that single reproducer run, the tracing drain still died, but the training process completed all 20 steps. The instrumentation lost data without exhausting a device-side resource needed by the workload. One clean run does not prove containment of the banned timestamp mode across every failure schedule, and that mode remained excluded. The production configuration, which omits latency timestamps and uses host-pinned buffers, has not produced a wedge in any acceptance, A/B, or soak run; the formal acceptance run and the five-day, eighteen-hour soak completed without one. The experiment also supported a general safety rule: when a tracing buffer can live outside scarce device memory, prefer the host and make record loss visible.

A second lesson came from our first watchdog. We tried to disable collection when too many buffers remained outstanding for too long. During a healthy checkpoint-resume phase, several ranks legitimately retained more buffers for more than seven minutes. The watchdog disabled collection on five of eight ranks; the training run itself was never affected, but those ranks lost device-timing telemetry until restart. In the real wedge, by contrast, the outstanding count froze below that threshold.

The signal was useful; the automatic action was not. We changed the default to observe and alert from outside the process. This is a recurring systems lesson: a local heuristic may be unable to distinguish a long valid phase from a deadlock, while a cluster-wide view can compare ranks, scheduler state, storage activity, and training progress.

How we built the production path

RidgeScope uses progressively stronger evidence rather than one universal collector.

1. Keep a broad, independent baseline

The node agent continuously collects device and fabric telemetry through DCGM and NVML: engine and SM activity, compute-pipe activity, memory, power, clocks, throttling, PCIe, NVLink, and hardware health. It attributes those samples to the scheduler allocation and the GPU process.

That baseline remains useful even when every deeper tier is absent. It also monitors the monitors. If CUPTI delivery stops while SM activity and kernel-launch behavior change on one rank, the independent layer exposes the sequence instead of accepting the in-process view as truth.

2. Use late-attach probes for continuous host semantics

Host-side probes observe CUDA launches, synchronization, allocation failures, memory copies, NCCL collective calls, communicator operations, call duration, and in-flight state. They can attach after a library is loaded and do not require application source changes.

This tier deliberately labels its durations as host-side. It is enough to detect that collectives stopped, one communicator initialization remained in flight, launch gaps expanded, or a process stopped reaching CUDA. It does not claim that an enqueue duration is GPU execution time.

3. Add CUPTI device timing only by client-controlled opt-in

For jobs that need device-side evidence, a small component is injected at process start through CUDA's supported injection mechanism. It loads the CUPTI library from the workload environment and enables only the activity subset that survived our version and workload matrix: concurrent kernel activity and CUPTI's own overhead records. It does not use the banned latency mode or the hardware-counter profiling tier.

The component reduces records to a bounded, versioned representation inside the process. Raw kernel names are limited; common work is classified into a fixed vocabulary; high-rate records are batched. The workload-side component does not open a network connection or export telemetry. The node agent harvests those batches from outside the process, adds job and rank identity, and performs the actual export.

This separation limits blast radius. A dead agent leaves a no-op probe behind. A failed CUPTI subscription leaves the component dormant. A record kind outside the expected set is dropped. None of those paths should change the application's return code.

4. Treat coverage and collector health as product data

We expose four distinct coverage stages: the job requested instrumentation, the component loaded, the node agent attached, and records are flowing. We also track delivery heartbeat, requested and completed buffers, missed batches, and dropped records.

Those signals prevent a dangerous inference. Zero collected kernels can mean the GPU executed no kernels, but it can also mean the job did not opt in, injection failed, the agent attached after a short process exited, the injected component remained dormant, or CUPTI stopped delivering. A collision that prevents RidgeScope's subscriber from starting is rare with process-start injection because the component normally subscribes during CUDA initialization, before later framework-profiler sessions. If it does occur, it can look like a dormant injected process; until that distinction has its own reliable signal, the product should expose the ambiguity rather than name one cause. A trustworthy dashboard says which stages it observed and stops where the evidence stops.

5. Prove the path with real training, not a toy alone

Microbenchmarks helped establish record exactness and callback cost. They did not exercise model load, CUDA graphs, distributed collectives, checkpoint pauses, framework profilers, or multi-hour state decay.

Our acceptance sequence ended with an eight-H100 training smoke test and then a multi-day soak. In the formal smoke run, all 20 steps completed and the exported kernel totals matched the per-rank CUPTI census across all eight ranks. The unresolved collision policy on single-subscriber runtimes is the first reason this tier remains opt-in rather than a default fleet promise: instrumentation that starts first can deny a later developer profiler. The deliberately record-heavy smoke also showed a 6.8–8.7% step-time increase. In the longer run, the collector observed roughly 9.4 billion kernel records per rank over five days and eighteen hours without reported record drops, while agent resource use remained stable.

These numbers are not universal overhead claims. They establish measured cost and durability for this H100 stack; the short onboarding smoke keeps that evidence specific when the device-timing tier moves to another runtime.

Ways around the NCCL visibility gaps

Many fleets cannot use the newest NCCL profiler interface everywhere. The practical answer is a source ladder with explicit semantics.

Use the official profiler when the runtime supports it

The NCCL profiler interface is the cleanest source of collective identity because the library already knows the communicator, rank, operation, message size, algorithm, protocol, and internal event hierarchy. Newer interfaces add GPU-generated timing and kernel events. NVIDIA's Inspector plugin is a useful reference for the shape of always-on collection, but the public deadlock, crash, and silent-overwrite record means it should not be read as a safety guarantee.

Deploy it only after confirming the NCCL release inside the training environment, the interface version exported by the plugin, the selected event mask, the plugin's buffer and export behavior, and coexistence with any workload-selected plugin. Start with application-thread collective events. Proxy, network, and kernel events produce better detail but execute on more sensitive paths and at a much higher rate.

Use host probes for compatibility and late attach

Stable NCCL entry points provide collective type, message shape, host duration, rate, and in-flight state across a much wider range of NCCL versions. They work for jobs that were not launched with a profiler plugin and can often attach while a run is already active.

The limitation must stay visible in the name and explanation of the data: this is call or enqueue time. It becomes powerful when paired with device telemetry. A host call that remains in flight while peer GPUs show engine-high, SM-low activity is strong evidence of a distributed wait even without exact device-side bandwidth.

Read progress passively when restart is impossible

For a long-running job that cannot be restarted, read-only sampling of NCCL's host-visible communicator state can answer a narrower but urgent question: are the GPU-consumed progress counters advancing? We validated this as research; it is not a shipped RidgeScope collection tier.

On a 23-hour-old, single-node eight-H100 run, per-rank progress sampled from outside the process agreed with host enqueue counts to 0.3% and separated healthy compute from a checkpoint pause within seconds. The longest healthy freeze was 3.0 seconds. That measured ceiling supports a candidate wedge alarm at about 10 seconds—roughly 60 times earlier than the framework's 600-second NCCL watchdog—but the induced-wedge stage was not run, so the wedge signature remains a prediction rather than an observed result. The multi-node proxy extension also remains unexercised because every communicator in this test stayed within one node.

The method depends on internal structure layouts. It must be keyed to an exact binary fingerprint, validated by magic values and invariants, and fail closed when any offset or relationship is unknown. It is a liveness source, not a portable NCCL API.

Use CUPTI kernels as a bridge, carefully

On NCCL versions below the profiler-interface timing floor, this is the device-timing bridge RidgeScope ships. CUPTI times the kernels that implement collectives. Their names often encode collective, algorithm, and protocol information, so bounded classification can recover useful categories while host probes supply message bytes.

Kernel names are version-scoped evidence rather than a public NCCL contract, so the acceptance process census-checks classification on each workload and measures the unclassified “other” residue. On the validated ZeRO-3 workload, that residue was 0.08% of kernel device time. On the busiest rank, NCCL kernels occupied about 74% of kernel device time, and per-collective maxima exposed a 1.29-second Broadcast spin-wait. Those measured checks turn naming drift into a visible acceptance result instead of a silent assumption. CUDA graphs, multi-kernel collectives, and persistent kernels still require careful correlation because kernel residency can include synchronization time.

Keep debug logs as configuration evidence

NCCL debug output is excellent for proving which interfaces, transports, devices, channels, algorithms, and network plugins were selected at initialization. It is less suitable as a high-rate performance stream. Logs may be disabled, sampled, rotated, or lost with the node, and they usually do not provide a complete device execution timeline.

Use startup logs to establish configuration truth. Use structured metrics or traces for continuous timing and progress.

A practical decision table

Question	Preferred source	Fallback	Important caveat
Is this GPU doing broad compute work?	DCGM SM and compute-pipe activity	Engine activity plus power and clocks	Activity is not model progress
Did a rank stop issuing CUDA work?	Host kernel-launch probes and launch gaps	Per-process GPU activity and rank comparison	A CUDA graph can reduce visible host launches
Is communicator initialization stuck?	Host NCCL communicator in-flight state	Startup logs and rank process state	No device kernel may exist yet
Did collectives stop?	Host collective rate, in-flight state, and rank comparison	NCCL profiler events; version-qualified passive progress	A long valid collective needs peer and baseline context
How long did communication occupy the GPU?	NCCL GPU events or CUPTI NCCL-kernel timing	DCGM link activity over the step	Kernel residency can include spin-wait
What bus bandwidth did a collective achieve?	Message bytes joined to GPU-side duration	Link throughput over a known collective window	Host enqueue duration is the wrong denominator
Why is a kernel slow internally?	Scheduled counter profile or PC sampling	Compare activity, memory, clocks, and peers	Do not run counter profiling blindly beside DCGM

The acceptance checklist we wish we had at the start

1. Fingerprint the libraries inside the rank. Record the loaded CUPTI and NCCL binaries, not only the host toolkit or container label.
2. Define the semantic contract of every duration. State whether it is host call, queue, device kernel, proxy, or end-to-end time.
3. Make non-coverage explicit. Separate opt-out, load failure, attach delay, no activity, and broken delivery.
4. Fail inert. Every initialization, parsing, allocation, and export failure must leave the workload able to continue.
5. Test tool collisions. Run with the framework profiler, Nsight tooling used by the team, DCGM profiling, CUDA graphs, and any existing NCCL plugin.
6. Use a matched control. Compare instrumented and uninstrumented runs on the same hardware and workload phase; report the distribution, not one step.
7. Exercise pressure. Test launch storms, large collectives, checkpoint pauses, long model initialization, buffer backpressure, and exporter outages.
8. Watch the observer externally. Heartbeat, buffer turnover, drops, queue age, agent resource use, and rank-level asymmetry belong in the monitoring surface.
9. Soak across job boundaries. Verify rediscovery, attach and detach, PID reuse, library replacement, cardinality, and state cleanup over days.
10. Gate every version change. A new CUDA or NCCL minor release is a new experiment until the workload-shaped suite passes.

What RidgeScope does with these layers

RidgeScope does not treat a CUPTI record or an NCCL call as a diagnosis. The on-node agent correlates them with scheduler lifecycle, GPU and fabric telemetry, host pressure, process state, storage behavior, and logs. The product then evaluates the evidence per job and per rank.

This matters because identical low-level signals can mean different things. A long NCCL kernel with balanced peer progress can be a legitimately large collective. The same kernel with one frozen rank, flat training progress, and engine-high/SM-low peers is a hang. A cold GPU during heavy writes is probably checkpointing; a cold GPU with a silent input queue is a different investigation.

The device-timing tier remains client-controlled. It needs no change to model source code, but it does require the job to be launched with instrumentation enabled. When that is not possible or not supported, RidgeScope continues with its late-attach host and fleet evidence and says what those signals can and cannot establish. That boundary is more valuable than a precise-looking number with an unknown provenance.

Production observability is not the maximum amount of data a profiler can collect. It is the maximum amount of trustworthy evidence the workload can safely afford.