Transformer Parameters Calculator
Estimate a transformer model's total parameter count from its layer count, hidden dimension, and vocabulary size.
Inputs
Used to verify head_dim = hidden_dim / heads is an integer.
Used only to report positional-embedding parameters for learned position encodings.
Total Parameters
6.57billion
Attention Parameters
2,147,483,648
Feed-Forward Parameters
4,294,967,296
Embedding Parameters
131,072,000
Per-Head Dimension
128.0
Step by step
Attention params: 4 × hidden² × layers (Q, K, V, output projections)
4 × 4096² × 32
= 2,147,483,648
Feed-forward params: 8 × hidden² × layers (4x expansion, up + down projections)
8 × 4096² × 32
= 4,294,967,296
Embedding params: vocab_size × hidden_dim
32,000 × 4096
= 131,072,000
Total: attention + feed-forward + embeddings
2,147,483,648 + 4,294,967,296 + 131,072,000
= 6,573,522,944 (6.57B)
How it works
A transformer's parameters are dominated by two per-layer blocks repeated across all layers: the attention block (Query, Key, Value, and output projections, each roughly hidden² in size, giving ≈4 × hidden² × layers) and the feed-forward block (typically a 4x expansion up-projection and a matching down-projection, giving ≈8 × hidden² × layers). Add the token embedding table (vocab_size × hidden_dim, often tied/shared with the output head) to get the model's total parameter count: total = attention + feed-forward + embeddings. This is the same scaling relationship used to estimate the size of GPT-style and LLaMA-style models from their published architecture configs.
Formula
total_params = 4 * hidden^2 * layers + 8 * hidden^2 * layers + vocab_size * hidden
- d
- Hidden dimension (model width)
- L
- Number of transformer layers
- V
- Vocabulary size
Frequently Asked Questions
Why is the feed-forward block roughly twice the size of attention?
The feed-forward network expands the hidden dimension by 4x and then projects back down, giving two matrices of size hidden × (4×hidden), for ≈8×hidden² total, versus attention's four hidden×hidden projection matrices (Q, K, V, output) totaling ≈4×hidden².
Are embedding and output head parameters double-counted?
Most modern LLMs tie (share) the input embedding matrix and the final output/unembedding layer, so this calculator counts the embedding table once; if your model does not tie weights, add vocab_size × hidden_dim again for a separate output head.
Why include sequence length and positional embeddings separately?
Many modern architectures use rotary (RoPE) or relative position encodings that add no extra parameters, while older architectures use learned absolute position embeddings that do — this figure is reported separately so you can include or exclude it as appropriate for your architecture.
How accurate is this against real published models?
This formula is a standard approximation and typically lands within a few percent of official parameter counts for dense (non-mixture-of-experts) decoder-only transformers; MoE models, grouped-query attention, and other efficiency tricks will shift the exact numbers.
You might also need
- Parameter Count CalculatorCommonly used together
- Multi-head Attention Parameters CalculatorCommonly used together
- Positional Encoding CalculatorCommonly used together
- LSTM Parameters CalculatorCommonly used together
- VRAM Usage CalculatorCommonly used together
- CNN Parameter Count CalculatorAlso in Neural Networks