KV Cache Memory Calculator
Calculate the memory required for the key-value cache during LLM inference.
Inputs
Total KV Cache Memory
2.147GB
Memory per Token
524.29KB
Total KV Cache Memory
2,147,483,648bytes
Step by step
KV cache memory: 2 × layers × hidden_dim × seq_len × batch × bytes/element
2 × 32 × 4096 × 4096 × 1 × 2
= 2,147,483,648 bytes
In GB: total bytes ÷ 1e9
2,147,483,648 ÷ 1e9
= 2.147 GB
Per-token cache size: total bytes ÷ (sequence length × batch size)
2,147,483,648 ÷ (4096 × 1)
= 524.29 KB/token
How it works
During autoregressive generation, transformers cache the key and value projections for every previous token so they don't need to be recomputed at each new decoding step. This memory grows with: memory = 2 × num_layers × hidden_dim × sequence_length × batch_size × bytes_per_element, where the factor of 2 accounts for storing both keys and values. KV cache memory scales linearly with context length and batch size, which is why long-context serving and high-throughput batching are major memory bottlenecks for LLM inference — often exceeding the memory used by the model's own weights at large batch sizes or long sequences.
Formula
kv_cache_bytes = 2 * num_layers * hidden_dim * sequence_length * batch_size * bytes_per_element
- L
- Number of transformer layers
- d
- Hidden dimension (model width)
- s
- Sequence length (tokens cached)
- b
- Batch size (concurrent sequences)
- B_e
- Bytes per element (2 for FP16, 4 for FP32, 1 for INT8)
Frequently Asked Questions
Why does the KV cache matter so much for LLM serving?
It's the memory cost of supporting long conversations and large batch sizes during inference — since it scales with sequence length × batch size, it can dwarf the model's own weight memory for long-context or high-throughput deployments.
How can I reduce KV cache memory?
Common techniques include multi-query attention (MQA) or grouped-query attention (GQA), which share key/value heads across multiple query heads, quantizing the cache to INT8, and using shorter context windows or cache eviction strategies.
Does hidden_dim here mean the full model hidden size?
It should represent the total key/value dimension across all attention heads for one layer (num_heads × head_dim), which for standard multi-head attention equals the model's hidden size.
How does batch size affect KV cache differently from model weight memory?
Model weights are loaded once regardless of batch size, but the KV cache must be allocated per sequence in the batch, so KV cache memory scales linearly with batch size while weight memory does not.