Estimate the VRAM consumed by model weights, activations, and optimizer states.
Training VRAM has four components: weights, gradients, optimizer states, and activations. With Adam in FP32, each holds roughly 1x model size (weights) + 1x (gradients) + 2x (Adam's momentum and variance buffers) = 4x model size before activations. Mixed-precision training stores FP16 working copies of weights/gradients for fast compute alongside an FP32 'master' copy of weights for stable updates (2+4=6 bytes/param for weights alone), which is why mixed precision reduces compute time but doesn't reduce total memory as much as switching everything to FP16 would. Activation memory scales linearly with batch size and is the most controllable lever for fitting training into limited VRAM.
total_vram = weights_memory + gradients_memory + optimizer_states_memory + activations_memory
Mixed precision keeps an FP32 'master' copy of weights (and often accumulates gradients/optimizer state in FP32) for numerical stability, so it adds memory on top of the FP16 working copies rather than simply replacing FP32 everywhere — the main benefit is faster compute, not a full 2x memory reduction.
AdaFactor approximates Adam's per-parameter second-moment statistics with a factored (row/column) representation instead of a full same-shape buffer, cutting optimizer state memory roughly in half or more compared to Adam's two full-size moment buffers.
Lower the batch size (activations scale linearly with it), use gradient checkpointing to trade compute for activation memory, switch to a lower-memory optimizer like AdaFactor or 8-bit Adam, or use gradient accumulation to simulate a larger batch without the memory cost.
No, the KV cache applies to autoregressive inference/generation, not training forward/backward passes — use the KV Cache Memory Calculator for that scenario.