From Storage to Tensor Core: How the Training Data Pipeline Feeds a GPU

A GPU cannot train on a file path. Before a sample reaches a matrix multiplication, storage returns bytes, the operating system moves or caches pages, CPU workers decode and transform records, a collator builds tensors, the training process receives a batch, host memory becomes suitable for DMA, and a copy engine moves the result into device memory.

The training loop hides that supply chain behind one iterator:

for batch in dataloader:
    batch = batch.to("cuda", non_blocking=True)
    output = model(batch)

If the chain stays ahead of the GPU, it disappears beneath compute. If one stage falls behind, the device runs a burst of kernels and then waits for the next delivery. The resulting sawtooth is often called “low GPU utilization,” but utilization is the final symptom. The cause may live in storage latency, page faults, tokenization, a multiprocessing queue, NUMA placement, pageable memory, PCIe, or a single slow rank's input shard.

The ordinary path has more copies than it appears to

For a conventional PyTorch DataLoader on an x86 GPU server, the common path looks like this:

That drawing is intentionally conventional. GPUDirect Storage can create a direct DMA path between supported storage and GPU memory, avoiding a CPU bounce buffer. It is not automatically activated by setting pin_memory=True, and a typical Python Dataset still reads and transforms data on CPUs. Direct I/O is most useful when the application's first meaningful consumer of the bytes already runs on the GPU.

1. Storage returns bytes, often through the page cache

A dataset may reside on local NVMe, a parallel filesystem, NFS, or an object store behind a client library. The relevant performance dimensions differ:

• Bandwidth limits large sequential reads.
• Latency and metadata operations dominate millions of small files.
• Concurrency determines whether workers can keep enough requests in flight.
• Contention changes as other jobs read checkpoints or datasets from the same service.

Ordinary buffered filesystem reads pass through the kernel's page cache. On the first epoch, a page may come from storage. On a later epoch, the same read may be satisfied from RAM. That makes the second epoch faster without any change to the dataset code or storage system.

This also complicates diagnosis. Low disk throughput does not prove that the input path is inactive; it may mean the cache is serving the data. High cache-hit rates do not prove the GPU is fed; CPU decoding after the read may still be slower than training.

Compressed formats move the bottleneck deliberately. Fewer bytes cross storage, but CPUs must decode them. Large packed shards reduce metadata operations, but a poor shuffle strategy can produce long seeks or correlated batches. An input format is part of the performance model, not merely a storage choice.

2. DataLoader workers manufacture batches

With num_workers=0, the training process calls Dataset.__getitem__(), runs transforms, collates the batch, and only then submits GPU work. CPU input preparation and GPU compute take turns.

Setting num_workers > 0 creates worker processes. For a map-style dataset, the main process usually decides which indices belong to each batch and sends those index tasks to workers. Each worker reads samples, applies dataset code and transforms, and runs the collation function. Completed batches return through a result queue.

Workers are processes rather than ordinary Python threads, so they can run Python work concurrently without sharing one Global Interpreter Lock. They also have separate address spaces. PyTorch warns that workers can eventually consume memory comparable to the parent process for Python objects they access. A parent that holds a huge Python list of filenames can therefore multiply host-memory use by the worker count.

prefetch_factor controls how many batches each worker prepares in advance. With the documented default of two when multiprocessing is enabled, four workers can have up to eight prefetched batches across the worker pool. This queue absorbs jitter. It also consumes host memory and can hide slow individual samples until the queue drains.

persistent_workers=True keeps worker processes and their dataset instances alive after one pass through the loader. It avoids process startup and dataset initialization between epochs, at the cost of retaining their memory and open resources.

3. Collation changes records into transfer-shaped tensors

Storage records rarely match the final batch tensor. Images are decoded, resized, cropped, normalized, and stacked. Text is read, tokenized, truncated or packed, masked, and padded. Variable-length samples become rectangular tensors through padding or sequence packing.

Collation affects both CPU cost and every downstream byte:

• Padding increases host memory, PCIe traffic, activation memory, and GPU work without adding training tokens.
• Many small tensors create more Python objects and more transfer calls than a few contiguous tensors.
• Converting dtype on the CPU changes transfer volume; converting on the GPU spends device work after transfer.
• Random augmentation can make batch-preparation latency highly variable even when average throughput looks sufficient.

The slowest batch matters more than the average if it arrives after the prefetch queue empties. A dataset with one expensive sample every few hundred records can produce periodic GPU gaps that disappear in a throughput average.

4. Multiprocessing needs a handoff

A worker cannot return a pointer to ordinary private memory and expect the training process to use it directly. PyTorch's multiprocessing machinery reduces tensors into shared-memory representations and passes metadata through queues. The main process reconstructs a view of the batch.

This handoff consumes file descriptors, shared-memory capacity, queue operations, and CPU time. Container environments with a small /dev/shm can fail or stall even when the node has abundant ordinary RAM. A slow consumer can leave completed batches queued, increasing memory pressure. A dead worker can leave the training process waiting for a result that will never arrive unless timeout and failure handling expose it.

The iterator boundary—when Python asks for the next batch—is therefore not a storage boundary. It is the point where the training process waits for the entire upstream chain to have produced a consumable batch.

5. Pinned memory makes the host buffer suitable for DMA

Normal pageable host memory can move, be reclaimed, or be paged out by the operating system. A GPU DMA engine needs a stable physical mapping while a transfer is in flight. If a CUDA copy starts from pageable memory, the runtime may first stage the data through a pinned buffer, adding a copy and reducing the opportunity for asynchronous overlap.

Pinned, or page-locked, memory stays resident and provides the high-bandwidth path described in NVIDIA's CUDA best-practices guide. DataLoader(pin_memory=True) asks PyTorch's pinning path to place recognized tensors in pinned memory before returning the batch.

Pinning is not free:

• It consumes physical RAM that the operating system cannot page out.
• Creating pinned buffers has overhead, so reusable pools and the DataLoader pinning path matter.
• Custom batch classes need a pin_memory() method or their tensors may remain pageable.
• Pinning too much can reduce memory available to the page cache and other jobs.

6. The usual CPU-to-GPU road is PCIe, not NVLink

On a conventional x86 GPU server, host-to-device data normally crosses PCIe. NVLink primarily connects GPUs to other GPUs. It accelerates peer transfers and collectives inside systems such as HGX, but a DataLoader batch in CPU DRAM does not automatically travel over NVLink because the GPUs have NVLink links.

There are important exceptions. Grace Hopper systems connect CPU and GPU memory through NVLink-C2C. GPUDirect Storage can route data from supported storage or NICs directly into GPU memory, and some topologies can use a peer GPU plus NVLink as an intermediate path. These are platform-specific data paths, not a generic property of batch.to("cuda").

PCIe performance depends on more than the advertised link generation:

• A GPU or NIC can negotiate below its maximum link width.
• A dual-socket host can place the worker's memory on a remote NUMA node, forcing traffic across the CPU interconnect before PCIe.
• Several GPUs can share an upstream PCIe switch or root complex.
• Many tiny copies pay more per-call overhead than one packed transfer.

NVIDIA's best-practices guidance therefore recommends batching small transfers and using pinned memory. A topology-aware process placement keeps the data workers, host memory, and target GPU near one another.

7. CUDA streams determine whether copying hides under compute

CUDA operations are asynchronous with respect to the host. A copy request and a kernel launch are queued into streams. Operations in one stream observe stream order. Operations in different streams may run concurrently when their dependencies and hardware resources allow it.

This produces three common schedules:

A naïve loop can still expose the copy even with non_blocking=True because the next compute operation depends immediately on the copied tensor. Double buffering prepares device storage for the next batch and places its copy in a suitable stream while the current batch computes. An event or stream dependency prevents the next compute from reading the buffer early.

Overlap is a latency-hiding technique, not infinite capacity. If preparing and transferring the next batch takes longer than computing the current one, some delivery time remains exposed.

8. HBM is the destination, not yet the computation

Once the batch reaches high-bandwidth device memory, CUDA kernels can consume it. The GPU's thread-block scheduler assigns kernel work to streaming multiprocessors. Deep-learning matrix multiplications can use Tensor Cores when their dtype, dimensions, layout, and library implementation support the relevant instructions.

A tensor being present in HBM does not guarantee Tensor Core activity. The next operation might be an embedding lookup, index operation, normalization, data conversion, small reduction, or memory-bound elementwise kernel. GPU engine activity can therefore be high while Tensor Core activity remains low.

This is an important boundary in the phrase “feeding the GPU.” The input pipeline is healthy when it makes the next batch available on time. Whether the model turns that batch into efficient matrix work is a separate compute question.

9. Data starvation has a recognizable timeline

An input-starved run often alternates between two states:

1. A prepared batch arrives, and the GPU launches a dense burst of forward and backward kernels.
2. The batch finishes, but the next iterator result is not ready. Kernel launches stop, SM activity falls, and the device waits.

The important signal is the gap between bursts. Averages can hide it. Millions of normal microsecond launch intervals outweigh a smaller number of hundred-millisecond or multi-second gaps in a percentile calculation, even though those rare gaps consume most wall time.

Queue depth is especially useful. An empty queue at the moment the GPU asks for a batch proves the upstream pipeline did not keep up. A consistently full queue rules out storage and worker preparation for that interval and moves the investigation toward pinning, transfer, or GPU-side work.

10. One slow input rank becomes a distributed straggler

Each data-parallel rank has its own input pipeline. The rank that receives the hardest-to-decode samples, a cold storage shard, a remote NUMA allocation, or a contended CPU can begin forward later than its peers.

Synchronous training converts that local delay into global idle time. Faster ranks eventually reach gradient collectives and wait for the slow rank. From a fleet average, all GPUs may still look busy: the slow rank computes while its peers spin or wait. Per-rank timing reveals the asymmetry.

This is why tuning a loader on one GPU is necessary but not sufficient. Distributed input performance has a tail. Measure batch-ready latency and step arrival across ranks, not only aggregate samples per second.

A disciplined tuning order

Changing every DataLoader knob at once can improve throughput without explaining why. A stage-by-stage procedure produces a reusable answer:

1. Measure the GPU gaps. Confirm that the device actually waits between kernel bursts.
2. Measure iterator wait. Time how long the training process waits for the next batch, without confusing asynchronous GPU time with CPU time.
3. Inspect queue depth and worker latency. Decide whether the batch was late before pinning.
4. Sweep num_workers. Increase until throughput plateaus or CPU, memory, storage, or shared-memory pressure becomes the next limit.
5. Test persistent workers and prefetch depth. Reduce epoch startup and absorb jitter, while watching host-memory growth.
6. Enable and verify pinning. Confirm custom batch types are actually pinned.
7. Inspect H2D copies. Pack small tensors, use non-blocking copies appropriately, and test whether transfers overlap compute.
8. Check NUMA and PCIe topology. Keep worker CPU, host memory, and target GPU local where possible.
9. Compare ranks. A single local regression can tax the complete distributed job.
10. Change the data representation last. Sharding, packing, caching, or GPU-side decoding can move a confirmed bottleneck rather than guessing at one.

The optimal worker count is workload-specific. More workers can improve parallelism, then flatten, then make performance worse through CPU contention, duplicated parent memory, random storage I/O, or queue overhead. The goal is not to maximize workers. It is to keep the next batch ready at acceptable host cost.

A GPU is starved when the next dependency arrives after the current work ends. The empty interval is created upstream, but billed downstream.

Where RidgeScope fits

RidgeScope sees the downstream shape without requiring a framework SDK: per-GPU SM and engine activity, CUDA kernel launches and their gap distribution, PCIe traffic, process attachment, per-rank skew, job file I/O, host pressure, and the scheduler's allocation map.

Those signals narrow the cause. A GPU-cold interval with heavy job reads or writes points toward the storage path. A single rank with long gaps points toward a rank-local worker, CPU, link, or shard. Uniform engine-high/SM-low behavior with advancing NCCL calls points toward communication instead of input. The final application-level distinction—whether tokenization, augmentation, collation, or a particular dataset record was slow—still belongs in a CPU profile or DataLoader trace.

The purpose of cross-layer telemetry is not to replace that profiler. It is to identify the right lane and the right rank before the investigation begins.