Work out gpu memory for inference instantly with clear inputs, formula shown and shareable results.
Inference needs only the weights, the KV cache for in-flight sequences, and runtime overhead for CUDA context, buffers and allocator fragmentation. There are no gradients or optimiser moments, which is why serving is roughly an order of magnitude cheaper than training. Quantisation directly scales the dominant term: int4 at 0.5 bytes per parameter turns a 14 GB fp16 model into about 3.5 GB.
Inference memory
weights = params x bytes per weight; total = (weights + KV cache) x (1 + overhead)
It grows with concurrent sequences and context length, and serving engines such as vLLM preallocate it as a pool. Undersizing it caps concurrency; oversizing it wastes memory that could hold a bigger model.
Modern schemes such as GPTQ and AWQ lose very little on most benchmarks, but degradation is task-dependent and worse for reasoning and code. Always evaluate before shipping.