VRAM Usage Calculator
Break down GPU VRAM usage for training a model, including weights, gradients, and optimizer states.
Inputs
Estimated activation memory consumed per training sample (model/sequence-length dependent).
Total VRAM Required
116.00GB
Weights Memory
42.00GB
Gradients Memory
14.00GB
Optimizer States Memory
56.00GB
Activation Memory
4.00GB
Step by step
Weights (FP16 + FP32 master)
7B × (2 + 4) bytes
= 42.00 GB
Gradients
7B × 2 bytes
= 14.00 GB
Optimizer states (adam, ×2)
7B × 4 bytes × 2
= 56.00 GB
Activations: per-sample memory × batch size
0.5 GB × 8
= 4.00 GB
Total VRAM: weights + gradients + optimizer states + activations
42.00 + 14.00 + 56.00 + 4.00
= 116.00 GB
How it works
Training VRAM has four components: weights, gradients, optimizer states, and activations. With Adam in FP32, each holds roughly 1x model size (weights) + 1x (gradients) + 2x (Adam's momentum and variance buffers) = 4x model size before activations. Mixed-precision training stores FP16 working copies of weights/gradients for fast compute alongside an FP32 'master' copy of weights for stable updates (2+4=6 bytes/param for weights alone), which is why mixed precision reduces compute time but doesn't reduce total memory as much as switching everything to FP16 would. Activation memory scales linearly with batch size and is the most controllable lever for fitting training into limited VRAM.
Formula
total_vram = weights_memory + gradients_memory + optimizer_states_memory + activations_memory
- weights_memory
- Model weights (6 bytes/param mixed, 4 bytes/param FP32)
- gradients_memory
- Gradient buffers (2 bytes/param mixed, 4 bytes/param FP32)
- optimizer_states_memory
- Optimizer state (e.g. Adam: 2 * 4 bytes/param in FP32)
- activations_memory
- activation_per_sample * batch_size
Frequently Asked Questions
Why doesn't mixed precision cut memory in half?
Mixed precision keeps an FP32 'master' copy of weights (and often accumulates gradients/optimizer state in FP32) for numerical stability, so it adds memory on top of the FP16 working copies rather than simply replacing FP32 everywhere — the main benefit is faster compute, not a full 2x memory reduction.
Why does AdaFactor use less memory than Adam?
AdaFactor approximates Adam's per-parameter second-moment statistics with a factored (row/column) representation instead of a full same-shape buffer, cutting optimizer state memory roughly in half or more compared to Adam's two full-size moment buffers.
How can I reduce total VRAM usage without changing the model?
Lower the batch size (activations scale linearly with it), use gradient checkpointing to trade compute for activation memory, switch to a lower-memory optimizer like AdaFactor or 8-bit Adam, or use gradient accumulation to simulate a larger batch without the memory cost.
Does this include the KV cache?
No, the KV cache applies to autoregressive inference/generation, not training forward/backward passes — use the KV Cache Memory Calculator for that scenario.