Multi-head Attention Parameters Calculator
Calculate the number of parameters in a multi-head self-attention block.
About Multi-head Attention Parameters Calculator
Multi-head attention parameters come from four learned projection matrices — Query, Key, Value, and the output projection — each of shape d_model × d_model, giving params = 4 × d_model² per layer, independent of the number of heads (heads only split d_model into smaller subspaces for parallel attention computation). Separately, the KV cache needed during autoregressive generation grows with memory = 2 × num_layers × d_model × sequence_length × bytes_per_element, since keys and values must be stored for every previous token across every layer — this is why long-context inference becomes memory-dominated even though attention itself has a fixed parameter count.
Formulas
Attention parameters
attention_params = 4 * d_model^2 * num_layers
- d
- Model dimension (d_model)
- L
- Number of transformer layers
KV cache memory
kv_cache = 2 * num_layers * d_model * seq_len * bytes_per_element
- L
- Number of layers
- d
- Model dimension
- s
- Sequence length
- B_e
- Bytes per element
Worked example
With the default values below, Multi-head Attention Parameters Calculator returns Total Attention Parameters of 2,147,483,648. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Model Dimension (d_model)accepts 1–100,000 dims
- 4,096 dims
- Number of Headsaccepts 1–1,000
- 32
- Number of Layersaccepts 1–1,000
- 32
- Sequence Lengthaccepts 1–10,000,000 tokens
- 4,096 tokens
- Precision (KV Cache)one of FP32 (4 bytes), FP16 / BF16 (2 bytes), INT8 (1 byte)
- FP16 / BF16 (2 bytes)
Results
- Total Attention Parameters
- 2,147,483,648
- Total KV Cache Memory
- 2.147 GB
- KV Cache per Token
- 524.29 KB
- Per-Head Dimension
- 128.0
Step by step
- 1Attention params per layer: 4 × d_model² (Q, K, V, output projections)= 67,108,864
- 2Total attention params: per-layer params × num layers= 2,147,483,648
- 3KV cache per token: 2 × layers × d_model × bytes/element= 524.29 KB/token
- 4Total KV cache: per-token cache × sequence length= 2.147 GB
Assumptions and limits
- The result is derived only from the 5 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 3 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
Why doesn't the number of heads change the parameter count?
Splitting d_model into multiple heads just reshapes the same Q/K/V/output projection matrices into parallel subspaces for computing attention — the total parameter count of those matrices (4 × d_model²) stays the same regardless of how many heads you split them into.
Why does the KV cache scale with sequence length but attention parameters don't?
Parameters are fixed, learned weights that don't change size regardless of input; the KV cache, by contrast, is runtime activation memory that must store one key and value vector per token per layer, so it necessarily grows as more tokens are processed.
How does grouped-query attention (GQA) change this calculation?
GQA shares key/value projections across groups of query heads, reducing the K/V projection parameter count and — more importantly — shrinking the KV cache proportionally to the number of KV head groups instead of the full head count, which is the primary reason GQA is used in modern LLMs.
Is the output projection always the same size as Q/K/V?
In standard multi-head attention, yes — the output projection maps the concatenated per-head outputs (which sum back to d_model) through another d_model × d_model matrix, matching the size of the Q, K, and V projections.