RidgeScope finds where allocated GPUs stop producing useful work — idle jobs, hangs, stragglers, data starvation, checkpoint I/O, fabric bottlenecks, and runs that finish without learning. Then it shows what the evidence supports and the next action.
Every GPU, every training run, and the estimated cost of non-productive time in one view. Open any slow or wasteful run to see why.
Engine activity alone can report a waiting GPU as busy. RidgeScope combines SM work, kernels, collectives, rank behavior, and training progress before it calls a GPU productive.
Turn paid GPU time into productive trainingIdle is not productiveBusy-wait is not productiveCompleted does not always mean usefulPer-rank evidence, zero code changesSlurm training, explainedTurn paid GPU time into productive trainingIdle is not productiveBusy-wait is not productiveCompleted does not always mean usefulPer-rank evidence, zero code changesSlurm training, explained
What utilization should mean
A rented GPU is not necessarily a productive GPU
RidgeScope measures whether allocated GPUs are doing useful model work and whether the training run is advancing toward its intended outcome.
01
Allocated but idle
The job holds expensive GPUs while waiting on a launcher, dataloader, deadlock, or empty workload.
02
Busy but not productive
Engine activity looks high while ranks spin in NCCL, wait on a straggler, or serialize kernel launches.
03
Completed but wasted
The scheduler reports success, but the model did not learn, resumed from scratch, or barely trained.
Commercial occupancy answers whether capacity is sold. RidgeScope answers whether the capacity already allocated to training is producing useful work — and why it is not.
H100 cost-of-waste calculator
Put a price on non-productive GPU time
Use a current neocloud H100 reference or enter your contracted rate. Replace the default waste percentage with what RidgeScope measures in your own fleet.
Published per GPU-hour
%
Cost exposed to non-productive GPU time$1,079,232per year
Per day$2,957
Per month$88,704
Non-productive GPU-hours / year280,320
Total GPU spend / day$11,827
128 GPUs × 24h × $3.85/h × 25% non-productive time
This estimates cost exposure, not guaranteed savings. RidgeScope measures where allocated time is idle, stalled, waiting, or otherwise not producing useful training work.
Simple fleet pricing
Spend cents to protect every GPU dollar
Standard managed coverage starts at $40 per covered GPU per month ($0.055 per covered GPU-hour). The effective rate steps down with fleet size, every core feature is included, and larger fleets use custom agreements.
Kernel launch counts, API durations, in-flight collectives — separation of idle hang from real compute.
Job logs
Loss trajectory, framework progress, Python tracebacks — attributed by rank, node, and process.
Slurm scheduler records
Job & step lifecycle, per-node epilogue/prolog exit codes — encode failure isn't read as a code fault.
GPU engine activity is never trusted alone. RidgeScope corroborates SM work, kernel launches, collective progress, rank behavior, and the training trajectory before it calls a run productive.
Your training telemetry is part of your model IP. We treat it that way.
RidgeScope is designed for environments where job output, model behavior, and cluster telemetry are proprietary. Choose a managed or private deployment so you control where telemetry is stored and where investigations run.
RidgeScope analyzes
GPU and runtime telemetry
Slurm scheduler records
Job output you authorize
Not required
—Training datasets
—Model weights or checkpoint contents
—Training source code
01
Isolation by organization
Each organization receives separate data stores, organization-scoped application access, and write-only ingestion credentials.
02
Identity-bound access
Signed organization identity is verified across the API, live connections, provisioning, and secrets boundaries.
03
Scoped credentials
Vault mediates access to data services so applications use organization-scoped credentials instead of shared database passwords.
04
Encrypted and private
Data is encrypted in transit and at rest. Data stores, secrets, and telemetry services remain on private networks.
Deployment choice
Deploy where your risk model requires
Keep the operational simplicity of a managed service or place the full data and investigation path inside your own environment.
Private data services, organization-scoped access, and encrypted storage without another platform for your team to operate.
Private deployment
Your cloud, on-premises, or air-gapped
Keep telemetry, secrets management, and investigation inference inside your security boundary.
From waste to action
Know what is wasting the allocation — and what to do next
confidence 97%nccl-hang
“Job 2260 is currently hung inside an NCCL collective and will not recover without intervention.”
Every claim cited to a metric and log.
Recommended action, not just a diagnosis.
Shows how to confirm the cause — no guessing.
How a verdict reads
Anatomy of a verdict
Every investigation ends in a three-part verdict — what the data shows, what it implies, and how to confirm the cause — plus a single failure family and a confidence score. Here is a real “done-but-bad” case: the run that succeeded and learned nothing.
family E1 · loss-divergenceevery claim cited
01
What the data shows
The run completed cleanly and the GPUs were healthy — but the loss rose, then dead-plateaued near 11.9, with the gradient norm trending to zero. Every infrastructure tool read “healthy” until the loss was read.
02
What it implies
Loss divergence — a training-correctness failure, not an infrastructure one. The whole allocation produced a model that learned nothing, and because it completed cleanly, nothing else flagged it. The tell is the shape of the loss.
03
How to confirm the cause
Inspect the learning-rate schedule, precision and overflow settings, and dataset/label validation around the divergence window. Those checks distinguish optimizer, numerical, and data-quality causes.
The third part turns uncertainty into an action: it names the next evidence needed to confirm the cause.
Where GPU-hours disappear
Find waste across the whole training lifecycle
RidgeScope catches idle and slow runs, infrastructure failures, distributed coordination problems, and “done-but-bad” training that completes cleanly while producing no useful model.
D
Throughput & utilization
GPU under-utilization, I/O save-storms, stragglers — one slow rank drags the whole multi-GPU allocation.
E
Training correctness
Loss divergence, silent resume-from-scratch (checkpoint never loaded), underestimated runs.
F
Cluster & scheduler
Node-down mid-run, requeue / preemption, timeout — a job fault vs. an infrastructure event.
G
Healthy baseline
Confirms a genuinely healthy run on positive evidence, not on the absence of errors.
+ 3 more families covered (Startup & config · GPU memory & hardware · Distributed coordination) — every mode is named and ruled in or out with counterfactuals. Browse the full failure catalog
Why we stand apart
From GPU activity to training outcome
Infrastructure tools tell you that a GPU is allocated or active. RidgeScope reads the run — from the loss trajectory to per-rank collectives — and explains whether that activity is producing useful training work.
RidgeScope capability comparison with Datadog GPU, DCGM + Grafana, and NVIDIA Mission Control
Capability
RidgeScope
Datadog GPU
DCGM + Grafana
NVIDIA Mission Control
GPU telemetry (utilization, ECC, thermals)
✓
✓
✓
✓
Reads the loss trajectory from job output
✓
−
−
−
Per-rank CUDA & NCCL tracing — no SDK, no code changes
Plain-English root-cause verdict with cited evidence
RidgeScope✓
Datadog GPU—
DCGM + Grafana—
NVIDIA Mission Control—
Separates a job fault from an infrastructure fault, per run
RidgeScope✓
Datadog GPU—
DCGM + Grafana—
NVIDIA Mission Control—
Shows how to confirm the cause when evidence is incomplete
RidgeScope✓
Datadog GPU—
DCGM + Grafana—
NVIDIA Mission Control—
Based on publicly documented capabilities of each product, July 2026.
Framework support
Frameworks & training stacks we recognize
The agent reads loss, gradient norm, learning rate, and checkpoints directly from each job's standard output, via a registry of framework-specific parsers.
Hugging Face Transformers
Full ecosystem incl. the fine-tuning & alignment stack.
TRLLLaMA-FactoryAxolotlDeepSeek
Megatron-LM
Full ecosystem incl. large-scale tensor and pipeline-parallel workloads.
PyTorch Lightning
Full trainer lifecycle, step- and epoch-level telemetry parsing.
Keras / TensorFlow
Callback-level and fit-loop telemetry for TF2 and Keras training runs.
Coverage comes from parsers, not from SDKs the team has to install.
PCI DSS & GDPR · On-prem / air-gapped · SSO / SAML · Per-tenant isolation · Local LLMs — telemetry never leaves your environment
FAQ
Questions, answered
What does RidgeScope mean by productive GPU utilization?
A GPU is productive when it is doing useful model work and the training run is advancing. RidgeScope combines SM activity, kernel launches, collective progress, rank behavior, step progress, and the loss trajectory. It does not equate an allocated GPU—or even high engine activity—with useful compute.
How is this different from Datadog or Grafana?
They read infrastructure health and hand you dashboards and a query language. RidgeScope correlates the training itself — the loss trajectory, per-rank CUDA/NCCL activity, scheduler records — and explains where GPU time stopped being productive. A run can be allocated, busy, and still waste compute; RidgeScope identifies why.
Do we need to change our training code?
No. A lightweight on-node agent collects everything — GPU telemetry, eBPF-level CUDA/NCCL traces, job stdout, Slurm records. No SDK, no code changes, no special logging sink, no access to your training code required.
How fast do we see value?
The agent installs in minutes across a Slurm-scheduled cluster. The first cited investigation lands within 30 minutes, and Mission Control begins showing which allocations are computing, waiting, idle, or unhealthy.
Can it run on-prem or air-gapped?
Yes. On-premise and air-gapped deployments are supported, with local LLMs so telemetry never leaves your environment. PCI DSS & GDPR, SSO/SAML, encryption at rest and in transit, and strict per-tenant isolation.
Which training stacks are recognized?
Hugging Face Transformers (including TRL, LLaMA-Factory, Axolotl), Megatron-LM, PyTorch Lightning, and Keras/TensorFlow — via a registry of framework-specific parsers, not SDKs. For fully custom training loops, RidgeScope reports the infrastructure evidence and identifies the additional progress signal needed to confirm learning.
What happens when evidence is incomplete?
RidgeScope does not guess when evidence is incomplete. It shows what is proven, what is ruled out, and the next evidence needed to confirm the cause.
Find the waste. Prove the cause. Make GPU time productive.
Get a free audit of your current GPU utilization. We'll identify wasted capacity and the clearest opportunities to improve it.
