KV Cache Memory Calculator
Calculate the memory required for the key-value cache during LLM inference.
About KV Cache Memory Calculator
During autoregressive generation, transformers cache the key and value projections for every previous token so they don't need to be recomputed at each new decoding step. This memory grows with: memory = 2 × num_layers × hidden_dim × sequence_length × batch_size × bytes_per_element, where the factor of 2 accounts for storing both keys and values. KV cache memory scales linearly with context length and batch size, which is why long-context serving and high-throughput batching are major memory bottlenecks for LLM inference — often exceeding the memory used by the model's own weights at large batch sizes or long sequences.
Formula
kv_cache_bytes = 2 * num_layers * hidden_dim * sequence_length * batch_size * bytes_per_element
- L
- Number of transformer layers
- d
- Hidden dimension (model width)
- s
- Sequence length (tokens cached)
- b
- Batch size (concurrent sequences)
- B_e
- Bytes per element (2 for FP16, 4 for FP32, 1 for INT8)
Worked example
With the default values below, KV Cache Memory Calculator returns Total KV Cache Memory of 2.147. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Number of Layersaccepts 1–1,000 layers
- 32 layers
- Hidden Dimensionaccepts 1–100,000 dims
- 4,096 dims
- Sequence Lengthaccepts 1–10,000,000 tokens
- 4,096 tokens
- Batch Sizeaccepts 1–100,000 sequences
- 1 sequences
- Precisionone of FP32 (4 bytes), FP16 / BF16 (2 bytes), INT8 (1 byte)
- FP16 / BF16 (2 bytes)
Results
- Total KV Cache Memory
- 2.147 GB
- Memory per Token
- 524.29 KB
- Total KV Cache Memory
- 2,147,483,648 bytes
Step by step
- 1KV cache memory: 2 × layers × hidden_dim × seq_len × batch × bytes/element= 2,147,483,648 bytes
- 2In GB: total bytes ÷ 1e9= 2.147 GB
- 3Per-token cache size: total bytes ÷ (sequence length × batch size)= 524.29 KB/token
Assumptions and limits
- The result is derived only from the 5 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 3 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
Why does the KV cache matter so much for LLM serving?
It's the memory cost of supporting long conversations and large batch sizes during inference — since it scales with sequence length × batch size, it can dwarf the model's own weight memory for long-context or high-throughput deployments.
How can I reduce KV cache memory?
Common techniques include multi-query attention (MQA) or grouped-query attention (GQA), which share key/value heads across multiple query heads, quantizing the cache to INT8, and using shorter context windows or cache eviction strategies.
Does hidden_dim here mean the full model hidden size?
It should represent the total key/value dimension across all attention heads for one layer (num_heads × head_dim), which for standard multi-head attention equals the model's hidden size.
How does batch size affect KV cache differently from model weight memory?
Model weights are loaded once regardless of batch size, but the KV cache must be allocated per sequence in the batch, so KV cache memory scales linearly with batch size while weight memory does not.