Memory internals
Where GPU Memory Goes During Training
A practical map of parameters, gradients, activations, Adam states, CUDA workspaces, allocator reserves, fragmentation, and misleading OOM charts.
A model that occupies 14 GB for inference may require several times that amount for training. The weights are only the first resident. Training adds gradients, optimizer history, saved activations, communication buffers, library workspaces, and an allocator that deliberately keeps freed blocks for reuse.
This is why parameter count alone does not answer “Will it fit?” It gives the floor for one category. The peak depends on the optimizer, precision policy, batch shape, attention implementation, distributed strategy, temporary kernels, and the exact point in the step at which several categories coexist.
The most useful mental model separates GPU memory into three layers:
- Live tensors that the training algorithm still needs.
- Reusable allocator reserves that PyTorch owns but no current tensor occupies.
- Non-PyTorch allocations such as the CUDA context, NCCL, and library workspaces.
A dashboard, torch.cuda.memory_allocated(), and torch.cuda.memory_reserved() observe different combinations of those layers. Their disagreement is expected.
The memory map at a glance
1. Parameters are the easy part to calculate
A parameter occupies its element count multiplied by its storage width. One billion FP32 parameters require about 4 GB in decimal units; BF16 or FP16 parameters require about 2 GB. A 7-billion-parameter model therefore needs roughly 28 GB for FP32 weights or 14 GB for 16-bit weights before training creates anything else.
The word stored matters. Autocast changes the precision used by selected operations; it does not necessarily convert the model's persistent parameters. In the common PyTorch AMP pattern, the parameters remain FP32 while many forward activations and compute kernels use FP16 or BF16. Other training systems explicitly store parameters in BF16 and may keep an FP32 master copy for stable updates. “Mixed precision” is therefore not a complete memory specification.
2. Gradients create another parameter-shaped resident
For a dense, fully trainable model, every trainable parameter eventually receives a gradient of the same shape. The gradient may use the parameter dtype or a higher-precision accumulation dtype, depending on the training implementation.
Gradients appear progressively during backward. By the end of backward, most or all of them coexist with the parameters. Calling optimizer.zero_grad() with the default behavior may fill existing gradient buffers with zeros. Calling it with set_to_none=True drops their references, allowing the next backward pass to create them as needed. This can reduce memory traffic and changes the lifetime visible in an allocator trace.
Distributed training adds a related category: gradient-reduction buckets. DDP packs gradients into contiguous buckets to overlap all-reduce with backward. With gradient_as_bucket_view=True, parameter gradients can become views into bucket storage after the first iteration, avoiding a separate full copy between gradient tensors and communication buckets. Without such sharing, bucket memory can temporarily add another substantial resident.
3. Optimizer state can outweigh the weights
SGD without momentum stores little state beyond the parameters and gradients. Momentum SGD adds one parameter-shaped velocity tensor. Adam and AdamW usually maintain two: the first moment and the second moment. Many implementations keep those moments in FP32 even when model computation uses a lower precision.
| Common recipe | Persistent bytes per trainable parameter | What is included |
|---|---|---|
| FP32 SGD | about 8 | FP32 parameter + FP32 gradient |
| FP32 momentum SGD | about 12 | parameter + gradient + velocity |
| FP32 Adam/AdamW | about 16 | parameter + gradient + two moment tensors |
| Typical autocast + FP32 Adam | often still about 16 | persistent FP32 state remains; many activations and operations become 16-bit |
| Explicit 16-bit parameters | recipe-dependent | may use 16-bit gradients, FP32 moments, and an optional FP32 master copy |
These figures describe dense unsharded state, not a universal promise. Fused optimizers, quantized states, master-weight policies, gradient release, ZeRO, and FSDP alter the accounting. The safe procedure is to list each stored tensor rather than infer memory from a label such as “BF16 training.”
The first optimizer step often allocates moment tensors lazily. That creates a characteristic failure: forward succeeds, backward succeeds, then the first optimizer.step() runs out of memory. The model did not suddenly grow; the persistent optimizer state finally appeared.
4. Activations are controlled by the shape of the work
Parameters scale with model size. Activations scale with the execution: microbatch size, sequence length, hidden width, layer count, and which intermediate values backward needs.
During the forward pass, autograd saves tensors required by backward. Common examples include layer inputs, attention probabilities or their compact substitutes, normalization statistics, dropout masks, and outputs of nonlinear functions. These tensors accumulate as the forward pass descends through the network. Backward consumes them in reverse and releases them when no longer needed.
This makes activation memory dynamic:
- Doubling the microbatch commonly doubles many activation tensors.
- Doubling sequence length doubles token-wise activations and can increase naïvely materialized attention matrices quadratically.
- Variable-length batches create different peaks even at the same sample count.
- Gradient accumulation increases the effective batch without retaining every microbatch's activations at once, provided each microbatch completes backward before the next.
Activation checkpointing reduces the saved set. PyTorch retains the inputs at selected boundaries and reruns the discarded forward region during backward. It spends additional compute to lower the activation peak. This is distinct from saving a durable model checkpoint to storage; the two features share a name but solve different problems.
5. Temporary buffers are real even when they have no model name
High-performance kernels frequently need workspace. cuBLAS, cuDNN, attention kernels, compilers, and fused optimizers may request temporary buffers to choose faster algorithms, stage reductions, or transform layouts. Their lifetime may be one operator, one graph capture, or the full process.
Outputs can also overlap with inputs in unintuitive ways. A forward operation may allocate its output before an earlier temporary becomes reusable. Backward can create a gradient output while the saved forward input is still live. The peak is determined by this overlap, not by the sum of the largest tensor in each category measured separately.
Distributed libraries add communicator state and internal buffers. NCCL may allocate memory that the PyTorch caching allocator does not fully describe. CUDA Graph capture can reserve stable addresses for replay. Compilation and autotuning can temporarily exercise several algorithm candidates before settling on one.
6. The peak moves through the step
The exact maximum varies by execution order. It may occur:
- near the end of forward, when most saved activations coexist;
- during backward, when large saved tensors, gradients, and a workspace overlap;
- on the first optimizer step, when moments are initialized;
- during a checkpoint, if state is gathered or copied into staging buffers;
- during compilation or autotuning, before the steady-state algorithm is selected.
Warm-up steps are therefore part of capacity testing. A single successful forward pass proves neither that backward fits nor that the optimizer and checkpoint path fit.
7. Allocated, reserved, and visible are different numbers
Repeated calls to cudaMalloc and cudaFree are expensive and can introduce synchronization. PyTorch uses a caching allocator: it requests larger segments from CUDA, divides them into blocks, and reuses freed blocks for later tensors.
This creates two important PyTorch measurements:
memory_allocated()reports memory occupied by live PyTorch tensors.memory_reserved()reports memory managed by the caching allocator, including reusable blocks not occupied by live tensors.
nvidia-smi observes the process from the driver side. Its number can include allocator reserves, the CUDA context, library allocations, graph pools, and other memory that memory_allocated() does not count. A higher driver reading does not by itself indicate a leak.
torch.cuda.empty_cache() releases unoccupied cached blocks so other applications and driver tools can see them as free. It cannot release live tensors. It does not increase the amount of memory available to PyTorch when the missing capacity is held by objects the program still references.
8. Fragmentation is about shape, not just total free bytes
An allocator may have enough free bytes in aggregate but no reusable block large enough for the next request. Suppose it holds free blocks of 600 MiB, 300 MiB, and 200 MiB separated by live allocations. Their total is 1.1 GiB, but a contiguous 800 MiB request cannot use them as one block.
PyTorch exposes allocator statistics such as inactive split bytes, allocation retries, and OOM counts through memory_stats() and memory_summary(). memory_snapshot() captures segments, blocks, and allocation history for deeper analysis. These tools answer whether memory is held by live tensors, cached as reusable blocks, or stranded in a shape the next request cannot use.
9. Why the graph can stay below capacity after an OOM
An OOM reports a failed request. Failed memory was never allocated, so it cannot appear as a higher used-memory sample. If a process holds 65 GiB and requests another 20 GiB on an 80 GiB card, the request fails while the observed resident remains near 65 GiB.
Sampling adds two more blind spots:
- A temporary allocation can rise and fall between 15- or 30-second samples.
- Failure handling can release tensors before the next sample arrives.
The CUDA OOM message and allocator peak counters therefore answer different questions from a dashboard. The error tells you the request that could not fit at the allocation site. max_memory_allocated() tells you the highest live PyTorch tensor footprint reached. The driver series tells you the process-level footprint at sampling instants. None should be forced to contradict the others.
10. Choose a remedy for the category that dominates
| Dominant category | Typical lever | Tradeoff |
|---|---|---|
| Saved activations | Smaller microbatch or sequence; activation checkpointing; memory-efficient attention | More steps, recomputation, or changed kernel constraints |
| Parameters and gradients | FSDP/ZeRO sharding; freeze layers; parameter-efficient fine-tuning | Communication, complexity, or reduced trainable capacity |
| Optimizer state | Shard or offload state; lower-precision optimizer; different optimizer | Transfer overhead or changed numerical behavior |
| Temporary workspace | Select a lower-workspace algorithm; reduce shape; avoid concurrent peaks | Potentially slower kernels |
| Fragmentation | Stabilize shapes; inspect allocator snapshot; tune allocator after evidence | Configuration can mask rather than fix live-tensor growth |
| Checkpoint staging | Shard saves; bound asynchronous saves; control host and device staging | Longer or more complex save path |
The order matters. Emptying the cache will not solve a live activation peak. Activation checkpointing will not solve unsharded Adam state. Sharding optimizer state will not repair an accidental tensor retained in a Python list. First identify the owner and phase, then change the corresponding lifetime or representation.
A minimum memory investigation
- Record the model's parameter count, storage dtype, trainable subset, and optimizer.
- Measure allocated and reserved memory after model creation, after forward, after backward, and after the first optimizer step.
- Reset and read peak allocator statistics around a representative step.
- Compare several batch and sequence shapes, including the largest expected shape.
- Capture an allocator summary or snapshot when reserved memory greatly exceeds allocated memory.
- Separate PyTorch tensor memory from the process total visible to the driver.
- Exercise checkpoint save and load paths before declaring the configuration safe.
The question is not only how much memory the model needs. It is which tensors must coexist at the most expensive instant of the step.
Where RidgeScope fits
RidgeScope records GPU framebuffer usage across the run and correlates it with CUDA activity, failures, training phase, rank, and hardware capacity. It can distinguish a model that never left startup from one that allocated most of the card and failed in a real training step. It also preserves the CUDA error and failed allocation evidence that a sampled memory curve cannot reconstruct.
Allocator-level snapshots still belong inside the training process when the problem is fragmentation or a retained tensor. Infrastructure telemetry answers which GPU, which rank, which window, and whether the failure was workload- or hardware-driven. Framework instrumentation then explains the individual allocation sites. The two views meet at the memory phase.
