Vocabulary Size Calculator

Estimate the vocabulary size needed for a tokenizer given corpus statistics.

About Vocabulary Size Calculator

Vocabulary size is the count of unique tokens a tokenizer or model recognizes, and the out-of-vocabulary (OOV) rate measures how often tokens in real text fall outside that vocabulary and must be mapped to an <unk> token — computed as unknown token occurrences divided by total tokens, times 100. A high OOV rate signals that the vocabulary is too small or poorly matched to the target domain, leading to information loss during tokenization.

Formula

oov_rate = (unknown_tokens / total_tokens) × 100

unknown_tokens: Number of token occurrences not in vocabulary
total_tokens: Total token occurrences in the corpus

Worked example

With the default values below, Vocabulary Size Calculator returns Vocabulary Size of 30,000. Change any input above and every figure here is recalculated in your browser.

Inputs used

Unique Tokens (Vocabulary)accepts 1–10,000,000 tokens: 30,000 tokens
Total Tokens in Corpusaccepts 1–1,000,000,000,000 tokens: 5,000,000 tokens
Unknown (OOV) Token Occurrencesaccepts 0–1,000,000,000,000 tokens: 25,000 tokens

Results

Vocabulary Size: 30,000 tokens
OOV Rate: 0.500 %
Vocabulary Coverage: 99.500 %

Step by step

1Vocabulary size= 30,000 tokens
2OOV rate= 0.500%
3Coverage rate= 99.500%

Assumptions and limits

The result is derived only from the 3 inputs listed above. Anything not among them is not part of the calculation.
In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
Figures are rounded for display to 3 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.

How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.

Frequently Asked Questions

What's a typical OOV rate for a well-trained tokenizer?

Modern subword tokenizers (BPE, WordPiece, SentencePiece) rarely produce true OOV tokens since they can decompose any word into known subword units, achieving near-0% OOV rates — high OOV rates are more common with word-level (not subword) tokenizers.

How does vocabulary size affect model size?

The embedding and output layers scale with vocabulary size, so a larger vocabulary increases the model's parameter count and memory footprint, though it may reduce sequence lengths for the same text.

Should I always maximize vocabulary size to minimize OOV?

No — larger vocabularies increase model size and can dilute training signal for rare tokens; subword tokenization is generally preferred over simply growing a word-level vocabulary indefinitely.