Calculate the memory required for the key-value cache during LLM inference.
During autoregressive generation, transformers cache the key and value projections for every previous token so they don't need to be recomputed at each new decoding step. This memory grows with: memory = 2 × num_layers × hidden_dim × sequence_length × batch_size × bytes_per_element, where the factor of 2 accounts for storing both keys and values. KV cache memory scales linearly with context length and batch size, which is why long-context serving and high-throughput batching are major memory bottlenecks for LLM inference — often exceeding the memory used by the model's own weights at large batch sizes or long sequences.
kv_cache_bytes = 2 * num_layers * hidden_dim * sequence_length * batch_size * bytes_per_element
It's the memory cost of supporting long conversations and large batch sizes during inference — since it scales with sequence length × batch size, it can dwarf the model's own weight memory for long-context or high-throughput deployments.
Common techniques include multi-query attention (MQA) or grouped-query attention (GQA), which share key/value heads across multiple query heads, quantizing the cache to INT8, and using shorter context windows or cache eviction strategies.
It should represent the total key/value dimension across all attention heads for one layer (num_heads × head_dim), which for standard multi-head attention equals the model's hidden size.
Model weights are loaded once regardless of batch size, but the KV cache must be allocated per sequence in the batch, so KV cache memory scales linearly with batch size while weight memory does not.