Embedding Layer Size Calculator
Calculate the parameter count and memory footprint of an embedding layer.
About Embedding Layer Size Calculator
An embedding layer is a lookup table of shape (vocabulary_size × embedding_dimension). Each token in the vocabulary has its own dense vector. For GPT-2's 50,257 vocab with 768-dim embeddings, that's 38.6M parameters taking ~73 MB in FP16.
Formula
Embedding Size
params = vocab_size × embedding_dim; memory = params × bytes_per_param
- vocab_size
- Number of unique tokens in vocabulary
- embedding_dim
- Dimensionality of each token vector
Worked example
With the default values below, Embedding Layer Size Calculator returns Parameters of 38,597,376. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Vocabulary Sizeminimum 1
- 50,257
- Embedding Dimensionminimum 1
- 768
- Precision (bytes)one of FP32 (4 bytes), FP16 (2 bytes)
- FP16 (2 bytes)
Results
- Parameters
- 38,597,376
- Memory
- 73.62 MB
Step by step
- 1Parameters: vocab × dim= 38,597,376
- 2Memory: params × bytes_per_param= 73.62 MB
Assumptions and limits
- The result is derived only from the 3 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 2 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
Are embedding parameters trainable?
Yes — embedding vectors are learned during training and encode semantic meaning. Similar words end up with similar embedding vectors (word2vec property).
Why do LLMs have such large embedding layers?
Large vocabularies (32K-100K+ tokens) combined with high embedding dimensions (4096+) mean embedding layers can have hundreds of millions of parameters — a significant fraction of smaller models.
Related Calculators
You might also need
- Parameter Count CalculatorCommonly used together
- Positional Encoding CalculatorCommonly used together
- Embedding Dimension CalculatorCommonly used together
- Trainable Parameters CalculatorAlso in Neural Networks
- GRU Parameters CalculatorAlso in Neural Networks
- RNN Parameters CalculatorAlso in Neural Networks