Skip to content
ENExplore Calculators

GPU Memory For Inference Calculator

Work out gpu memory for inference instantly with clear inputs, formula shown and shareable results.

About GPU Memory For Inference Calculator

Inference needs only the weights, the KV cache for in-flight sequences, and runtime overhead for CUDA context, buffers and allocator fragmentation. There are no gradients or optimiser moments, which is why serving is roughly an order of magnitude cheaper than training. Quantisation directly scales the dominant term: int4 at 0.5 bytes per parameter turns a 14 GB fp16 model into about 3.5 GB.

Formula

Inference memory

weights = params x bytes per weight; total = (weights + KV cache) x (1 + overhead)

Worked example

With the default values below, GPU Memory For Inference Calculator returns GPU memory required to serve of 19.59. Change any input above and every figure here is recalculated in your browser.

Inputs used

Model parametersminimum 1
7,000,000,000
Bytes per weightaccepts 0.25–4
2
KV cache reservedminimum 0 GB
4 GB
Runtime and fragmentation overheadaccepts 0–100 %
15 %

Results

GPU memory required to serve
19.59 GB
Weights
13.04 GB
Runtime overhead
2.56 GB
Hardware fit
Fits a 24 GB GPU

Step by step

  1. 1Values used
  2. 2Inference memory
  3. 3GPU memory required to serve= 19.59 GB
  4. 4Weights= 13.04 GB
  5. 5Runtime overhead= 2.56 GB
  6. 6Hardware fit= Fits a 24 GB GPU

Assumptions and limits

  • The result is derived only from the 4 inputs listed above. Anything not among them is not part of the calculation.
  • In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
  • Figures are rounded for display to 2 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.

How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.

Frequently Asked Questions

Why reserve KV cache separately?

It grows with concurrent sequences and context length, and serving engines such as vLLM preallocate it as a pool. Undersizing it caps concurrency; oversizing it wastes memory that could hold a bigger model.

Does int4 quantisation hurt quality?

Modern schemes such as GPTQ and AWQ lose very little on most benchmarks, but degradation is task-dependent and worse for reasoning and code. Always evaluate before shipping.

Related Calculators

GPU Memory For Inference Calculator | Calcrivo