VRAM Usage Calculator
Estimate the VRAM consumed by model weights, activations, and optimizer states.
About VRAM Usage Calculator
Training VRAM has four components: weights, gradients, optimizer states, and activations. With Adam in FP32, each holds roughly 1x model size (weights) + 1x (gradients) + 2x (Adam's momentum and variance buffers) = 4x model size before activations. Mixed-precision training stores FP16 working copies of weights/gradients for fast compute alongside an FP32 'master' copy of weights for stable updates (2+4=6 bytes/param for weights alone), which is why mixed precision reduces compute time but doesn't reduce total memory as much as switching everything to FP16 would. Activation memory scales linearly with batch size and is the most controllable lever for fitting training into limited VRAM.
Formula
total_vram = weights_memory + gradients_memory + optimizer_states_memory + activations_memory
- weights_memory
- Model weights (6 bytes/param mixed, 4 bytes/param FP32)
- gradients_memory
- Gradient buffers (2 bytes/param mixed, 4 bytes/param FP32)
- optimizer_states_memory
- Optimizer state (e.g. Adam: 2 * 4 bytes/param in FP32)
- activations_memory
- activation_per_sample * batch_size
Worked example
With the default values below, VRAM Usage Calculator returns Total VRAM Required of 116.00. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Model Parametersaccepts 0.001–2,000 billion
- 7 billion
- Training Precisionone of FP32 (full precision), Mixed Precision (FP16 + FP32 master)
- Mixed Precision (FP16 + FP32 master)
- Optimizerone of Adam / AdamW, SGD, AdaFactor
- Adam / AdamW
- Batch Sizeaccepts 1–100,000 samples
- 8 samples
- Activation Memory per Sampleaccepts 0–1,000 GB
- 0.5 GB
Results
- Total VRAM Required
- 116.00 GB
- Weights Memory
- 42.00 GB
- Gradients Memory
- 14.00 GB
- Optimizer States Memory
- 56.00 GB
- Activation Memory
- 4.00 GB
Step by step
- 1Weights (FP16 + FP32 master)= 42.00 GB
- 2Gradients= 14.00 GB
- 3Optimizer states (adam, ×2)= 56.00 GB
- 4Activations: per-sample memory × batch size= 4.00 GB
- 5Total VRAM: weights + gradients + optimizer states + activations= 116.00 GB
Assumptions and limits
- The result is derived only from the 5 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, fees or charges or compounding frequency into account.
- Figures are rounded for display to 2 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
Why doesn't mixed precision cut memory in half?
Mixed precision keeps an FP32 'master' copy of weights (and often accumulates gradients/optimizer state in FP32) for numerical stability, so it adds memory on top of the FP16 working copies rather than simply replacing FP32 everywhere — the main benefit is faster compute, not a full 2x memory reduction.
Why does AdaFactor use less memory than Adam?
AdaFactor approximates Adam's per-parameter second-moment statistics with a factored (row/column) representation instead of a full same-shape buffer, cutting optimizer state memory roughly in half or more compared to Adam's two full-size moment buffers.
How can I reduce total VRAM usage without changing the model?
Lower the batch size (activations scale linearly with it), use gradient checkpointing to trade compute for activation memory, switch to a lower-memory optimizer like AdaFactor or 8-bit Adam, or use gradient accumulation to simulate a larger batch without the memory cost.
Does this include the KV cache?
No, the KV cache applies to autoregressive inference/generation, not training forward/backward passes — use the KV Cache Memory Calculator for that scenario.