Skip to content
ENExplore Calculators

Transformer Parameters Calculator

Estimate the total parameter count of a transformer model from its architecture config.

About Transformer Parameters Calculator

A transformer's parameters are dominated by two per-layer blocks repeated across all layers: the attention block (Query, Key, Value, and output projections, each roughly hidden² in size, giving ≈4 × hidden² × layers) and the feed-forward block (typically a 4x expansion up-projection and a matching down-projection, giving ≈8 × hidden² × layers). Add the token embedding table (vocab_size × hidden_dim, often tied/shared with the output head) to get the model's total parameter count: total = attention + feed-forward + embeddings. This is the same scaling relationship used to estimate the size of GPT-style and LLaMA-style models from their published architecture configs.

Formula

total_params = 4 * hidden^2 * layers + 8 * hidden^2 * layers + vocab_size * hidden

d
Hidden dimension (model width)
L
Number of transformer layers
V
Vocabulary size

Worked example

With the default values below, Transformer Parameters Calculator returns Total Parameters of 6.57. Change any input above and every figure here is recalculated in your browser.

Inputs used

Number of Layersaccepts 1–1,000 layers
32 layers
Hidden Dimensionaccepts 1–100,000 dims
4,096 dims
Number of Attention Headsaccepts 1–1,000
32
Vocabulary Sizeaccepts 1–10,000,000 tokens
32,000 tokens
Max Sequence Lengthaccepts 1–10,000,000 tokens
4,096 tokens

Results

Total Parameters
6.57 billion
Attention Parameters
2,147,483,648
Feed-Forward Parameters
4,294,967,296
Embedding Parameters
131,072,000
Per-Head Dimension
128.0

Step by step

  1. 1Attention params: 4 × hidden² × layers (Q, K, V, output projections)= 2,147,483,648
  2. 2Feed-forward params: 8 × hidden² × layers (4x expansion, up + down projections)= 4,294,967,296
  3. 3Embedding params: vocab_size × hidden_dim= 131,072,000
  4. 4Total: attention + feed-forward + embeddings= 6,573,522,944 (6.57B)

Assumptions and limits

  • The result is derived only from the 5 inputs listed above. Anything not among them is not part of the calculation.
  • In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
  • Figures are rounded for display to 2 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.

How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.

Frequently Asked Questions

Why is the feed-forward block roughly twice the size of attention?

The feed-forward network expands the hidden dimension by 4x and then projects back down, giving two matrices of size hidden × (4×hidden), for ≈8×hidden² total, versus attention's four hidden×hidden projection matrices (Q, K, V, output) totaling ≈4×hidden².

Are embedding and output head parameters double-counted?

Most modern LLMs tie (share) the input embedding matrix and the final output/unembedding layer, so this calculator counts the embedding table once; if your model does not tie weights, add vocab_size × hidden_dim again for a separate output head.

Why include sequence length and positional embeddings separately?

Many modern architectures use rotary (RoPE) or relative position encodings that add no extra parameters, while older architectures use learned absolute position embeddings that do — this figure is reported separately so you can include or exclude it as appropriate for your architecture.

How accurate is this against real published models?

This formula is a standard approximation and typically lands within a few percent of official parameter counts for dense (non-mixture-of-experts) decoder-only transformers; MoE models, grouped-query attention, and other efficiency tricks will shift the exact numbers.

Related Calculators

Transformer Parameters Calculator | Calcrivo