Estimate the GPU memory required to train or run a given model architecture.
GPU memory for inference is dominated by model weights: memory_GB = parameters_billions × bytes_per_param × overhead_multiplier. FP16/BF16 halves memory versus FP32, while INT8 and INT4 quantization shrink it further at some accuracy cost. The overhead multiplier (commonly ~20%) accounts for activations, the KV cache, and framework/runtime overhead that sit on top of raw weight storage. Training memory is substantially higher because it also needs gradients and optimizer states.
total_memory_GB = parameters_billions * bytes_per_param * (1 + overhead_percent / 100)
No, this estimates inference-only memory. Training requires roughly 4x more memory per parameter to also store gradients and optimizer states (e.g. Adam's two moment buffers).
Lower-precision formats store each parameter in fewer bytes, directly reducing memory. INT4 quantization can shrink a model to 1/8th the size of FP32 with some quality tradeoff.
20% is a reasonable default for short-context inference. Long-context workloads with a large KV cache may need 40-100%+ overhead — use the KV Cache Memory Calculator for a more precise figure.
Batch size mainly affects activation and KV cache memory, not the base weight memory calculated here. Larger batches increase the effective overhead needed.