GPU Memory Calculator
Estimate the VRAM required to run inference on a model given its parameter count and numeric precision.
Inputs
Extra memory for activations, KV cache, and framework overhead beyond raw weights.
Total VRAM Needed
16.80GB
Base Weight Memory
14.00GB
Bytes per Parameter
2.00bytes
Step by step
Base weight memory: parameters × bytes per parameter
7B × 2 bytes
= 14.00 GB
Total memory with overhead: base memory × (1 + overhead%)
14.00 GB × 1.20
= 16.80 GB
How it works
GPU memory for inference is dominated by model weights: memory_GB = parameters_billions × bytes_per_param × overhead_multiplier. FP16/BF16 halves memory versus FP32, while INT8 and INT4 quantization shrink it further at some accuracy cost. The overhead multiplier (commonly ~20%) accounts for activations, the KV cache, and framework/runtime overhead that sit on top of raw weight storage. Training memory is substantially higher because it also needs gradients and optimizer states.
Formula
total_memory_GB = parameters_billions * bytes_per_param * (1 + overhead_percent / 100)
- parameters_billions
- Model parameter count in billions
- bytes_per_param
- Bytes per parameter (4 for FP32, 2 for FP16, 1 for INT8, 0.5 for INT4)
- overhead_percent
- Percentage overhead for activations and KV cache
Frequently Asked Questions
Does this include training memory (gradients, optimizer states)?
No, this estimates inference-only memory. Training requires roughly 4x more memory per parameter to also store gradients and optimizer states (e.g. Adam's two moment buffers).
Why does precision matter so much?
Lower-precision formats store each parameter in fewer bytes, directly reducing memory. INT4 quantization can shrink a model to 1/8th the size of FP32 with some quality tradeoff.
What's a safe overhead percentage to use?
20% is a reasonable default for short-context inference. Long-context workloads with a large KV cache may need 40-100%+ overhead — use the KV Cache Memory Calculator for a more precise figure.
Does batch size affect this estimate?
Batch size mainly affects activation and KV cache memory, not the base weight memory calculated here. Larger batches increase the effective overhead needed.