04
Training observability
The GPU Training Failures That Hide From Your Dashboards
Why GPU-Util can look healthy during checkpoint stalls, NCCL hangs, stragglers, fabric downgrades, and thermal throttling—and which signals expose them.
18 min read →RidgeScope Journal
Field notes / Issue 01
What the training loop leaves out.
Low-level guides for ML engineers: follow the bytes, kernels, gradients, collectives, and memory allocations that turn a few lines of Python into a distributed training run.
01
Distributed steps
02GPU memory
03Input pipelines
04Hidden failures
Start with the machinery
Three foundations for reading a training run
03
Input pipeline
From Storage to Tensor Core: How the Training Data Pipeline Feeds a GPU
Follow training data from storage and the page cache through DataLoader workers, pinned memory, PCIe, HBM, CUDA kernels, and Tensor Cores.
17 min read →02
Memory internals
Where GPU Memory Goes During Training
A practical map of parameters, gradients, activations, Adam states, CUDA workspaces, allocator reserves, fragmentation, and misleading OOM charts.
15 min read →01
Anatomy of a step
What Actually Happens During One Distributed Training Step?
Trace a batch through data loading, CUDA transfers, forward and backward passes, DDP gradient buckets, optimizer updates, and checkpoints.
15 min read →
