Estimate the total parameter count of a transformer model from its architecture config.
A transformer's parameters are dominated by two per-layer blocks repeated across all layers: the attention block (Query, Key, Value, and output projections, each roughly hidden² in size, giving ≈4 × hidden² × layers) and the feed-forward block (typically a 4x expansion up-projection and a matching down-projection, giving ≈8 × hidden² × layers). Add the token embedding table (vocab_size × hidden_dim, often tied/shared with the output head) to get the model's total parameter count: total = attention + feed-forward + embeddings. This is the same scaling relationship used to estimate the size of GPT-style and LLaMA-style models from their published architecture configs.
total_params = 4 * hidden^2 * layers + 8 * hidden^2 * layers + vocab_size * hidden
The feed-forward network expands the hidden dimension by 4x and then projects back down, giving two matrices of size hidden × (4×hidden), for ≈8×hidden² total, versus attention's four hidden×hidden projection matrices (Q, K, V, output) totaling ≈4×hidden².
Most modern LLMs tie (share) the input embedding matrix and the final output/unembedding layer, so this calculator counts the embedding table once; if your model does not tie weights, add vocab_size × hidden_dim again for a separate output head.
Many modern architectures use rotary (RoPE) or relative position encodings that add no extra parameters, while older architectures use learned absolute position embeddings that do — this figure is reported separately so you can include or exclude it as appropriate for your architecture.
This formula is a standard approximation and typically lands within a few percent of official parameter counts for dense (non-mixture-of-experts) decoder-only transformers; MoE models, grouped-query attention, and other efficiency tricks will shift the exact numbers.