Multi-head Attention Parameters Calculator
Calculate the number of parameters in a multi-head self-attention block.
Inputs
Total Attention Parameters
2,147,483,648
Total KV Cache Memory
2.147GB
KV Cache per Token
524.29KB
Per-Head Dimension
128.0
Step by step
Attention params per layer: 4 × d_model² (Q, K, V, output projections)
4 × 4096²
= 67,108,864
Total attention params: per-layer params × num layers
67,108,864 × 32
= 2,147,483,648
KV cache per token: 2 × layers × d_model × bytes/element
2 × 32 × 4096 × 2
= 524.29 KB/token
Total KV cache: per-token cache × sequence length
524.29 KB × 4096
= 2.147 GB
How it works
Multi-head attention parameters come from four learned projection matrices — Query, Key, Value, and the output projection — each of shape d_model × d_model, giving params = 4 × d_model² per layer, independent of the number of heads (heads only split d_model into smaller subspaces for parallel attention computation). Separately, the KV cache needed during autoregressive generation grows with memory = 2 × num_layers × d_model × sequence_length × bytes_per_element, since keys and values must be stored for every previous token across every layer — this is why long-context inference becomes memory-dominated even though attention itself has a fixed parameter count.
Formulas
Attention parameters
attention_params = 4 * d_model^2 * num_layers
- d
- Model dimension (d_model)
- L
- Number of transformer layers
KV cache memory
kv_cache = 2 * num_layers * d_model * seq_len * bytes_per_element
- L
- Number of layers
- d
- Model dimension
- s
- Sequence length
- B_e
- Bytes per element
Frequently Asked Questions
Why doesn't the number of heads change the parameter count?
Splitting d_model into multiple heads just reshapes the same Q/K/V/output projection matrices into parallel subspaces for computing attention — the total parameter count of those matrices (4 × d_model²) stays the same regardless of how many heads you split them into.
Why does the KV cache scale with sequence length but attention parameters don't?
Parameters are fixed, learned weights that don't change size regardless of input; the KV cache, by contrast, is runtime activation memory that must store one key and value vector per token per layer, so it necessarily grows as more tokens are processed.
How does grouped-query attention (GQA) change this calculation?
GQA shares key/value projections across groups of query heads, reducing the K/V projection parameter count and — more importantly — shrinking the KV cache proportionally to the number of KV head groups instead of the full head count, which is the primary reason GQA is used in modern LLMs.
Is the output projection always the same size as Q/K/V?
In standard multi-head attention, yes — the output projection maps the concatenated per-head outputs (which sum back to d_model) through another d_model × d_model matrix, matching the size of the Q, K, and V projections.