Skip to content
ENExplore Calculators

Embedding Dimension Choice Calculator

Work out embedding dimension choice instantly with clear inputs, formula shown and shareable results.

About Embedding Dimension Choice Calculator

An embedding table holds cardinality times dimension parameters, so the dimension choice directly sets memory and the number of weights that must be learned per category. Common heuristics grow the dimension sub-linearly with cardinality — the fast.ai rule uses 1.6 x c^0.56 capped at 600 — because rare categories cannot support many parameters no matter how large the vocabulary.

Formula

Embedding sizing

fast.ai: d = min(cap, round(1.6 x c^0.56)); parameters = c x d; bytes = parameters x 4 in fp32

Worked example

With the default values below, Embedding Dimension Choice Calculator returns Recommended embedding dimensions of 512. Change any input above and every figure here is recalculated in your browser.

Inputs used

Distinct values / vocabulary sizeminimum 2
50,000
Heuristicone of fast.ai: min(600, round(1.6 x c^0.56)), Fourth root: c^0.25, TensorFlow: c^0.25 x 1.6
fast.ai: min(600, round(1.6 x c^0.56))
Hard cap on dimensionsaccepts 2–4,096
512

Results

Recommended embedding dimensions
512
Parameters in the embedding table
25,600,000
Embedding table size in fp32
97.66 MiB
Compression vs one-hot encoding
97.66 x

Step by step

  1. 1Values used
  2. 2Embedding sizing
  3. 3Recommended embedding dimensions= 512
  4. 4Parameters in the embedding table= 25,600,000
  5. 5Embedding table size in fp32= 102,400,000
  6. 6Compression vs one-hot encoding= 97.66 x

Assumptions and limits

  • The result is derived only from the 3 inputs listed above. Anything not among them is not part of the calculation.
  • In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
  • Figures are rounded for display to 2 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.

How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.

Frequently Asked Questions

Is a larger dimension always better?

No. Beyond the point where each category has enough examples to fit its vector, extra dimensions add memory and overfitting without improving accuracy.

How should very rare categories be handled?

Bucket them into a shared unknown token or hash them. Giving a category seen twice its own 128-dimensional vector guarantees noise.

Related Calculators

Embedding Dimension Choice Calculator | Calcrivo