Skip to content
ENExplore Calculators

Model Parameter Count Calculator

Work out model parameter count instantly with clear inputs, formula shown and shareable results.

About Model Parameter Count Calculator

A transformer block holds four d-by-d attention projections (query, key, value and output), giving 4d^2, plus a feed-forward network of two matrices d-by-md and md-by-d, giving 2md^2. With the usual expansion factor of 4 that is 12d^2 per layer. Adding the vocab-by-d embedding table reproduces published parameter counts closely, since layer norms and biases contribute well under one percent.

Formula

Transformer parameter count

per layer = 4d^2 (attention) + 2 m d^2 (FFN); total = layers x per layer + vocab x d

d
Hidden dimension
m
FFN expansion factor

Worked example

With the default values below, Model Parameter Count Calculator returns Total parameters of 6,573,522,944. Change any input above and every figure here is recalculated in your browser.

Inputs used

Transformer layersminimum 1
32
Hidden dimension (d_model)minimum 8
4,096
Feed-forward expansion factoraccepts 1–16
4
Vocabulary sizeminimum 1
32,000

Results

Total parameters
6,573,522,944
Parameters per transformer block
201,326,592
Token embedding parameters
131,072,000
Attention projection parameters
2,147,483,648

Step by step

  1. 1Values used
  2. 2Transformer parameter count
  3. 3Total parameters= 6,573,522,944
  4. 4Parameters per transformer block= 201,326,592
  5. 5Token embedding parameters= 131,072,000
  6. 6Attention projection parameters= 2,147,483,648

Assumptions and limits

  • The result is derived only from the 4 inputs listed above. Anything not among them is not part of the calculation.
  • In particular, this calculator does not take inflation, tax or compounding frequency into account.
  • Figures are rounded for display to whole numbers, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.

How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.

Frequently Asked Questions

Why is the embedding sometimes counted twice?

Models with an untied output head have a second vocab-by-d matrix. Weight tying reuses the input embedding and saves those parameters.

Does grouped-query attention change the count?

Yes. Sharing key and value projections across query heads shrinks two of the four projections, so attention parameters fall below 4d^2.

Related Calculators

Model Parameter Count Calculator | Calcrivo