Calculate the number of parameters in a multi-head self-attention block.
Multi-head attention parameters come from four learned projection matrices — Query, Key, Value, and the output projection — each of shape d_model × d_model, giving params = 4 × d_model² per layer, independent of the number of heads (heads only split d_model into smaller subspaces for parallel attention computation). Separately, the KV cache needed during autoregressive generation grows with memory = 2 × num_layers × d_model × sequence_length × bytes_per_element, since keys and values must be stored for every previous token across every layer — this is why long-context inference becomes memory-dominated even though attention itself has a fixed parameter count.
Attention parameters
attention_params = 4 * d_model^2 * num_layers
KV cache memory
kv_cache = 2 * num_layers * d_model * seq_len * bytes_per_element
Splitting d_model into multiple heads just reshapes the same Q/K/V/output projection matrices into parallel subspaces for computing attention — the total parameter count of those matrices (4 × d_model²) stays the same regardless of how many heads you split them into.
Parameters are fixed, learned weights that don't change size regardless of input; the KV cache, by contrast, is runtime activation memory that must store one key and value vector per token per layer, so it necessarily grows as more tokens are processed.
GQA shares key/value projections across groups of query heads, reducing the K/V projection parameter count and — more importantly — shrinking the KV cache proportionally to the number of KV head groups instead of the full head count, which is the primary reason GQA is used in modern LLMs.
In standard multi-head attention, yes — the output projection maps the concatenated per-head outputs (which sum back to d_model) through another d_model × d_model matrix, matching the size of the Q, K, and V projections.