Vocabulary Size Calculator

Calculate vocabulary size from unique tokens and the out-of-vocabulary (OOV) rate for unknown tokens.

Inputs

Unique Tokens (Vocabulary)tokens

Total Tokens in Corpustokens

Unknown (OOV) Token Occurrencestokens

Vocabulary Size

30,000tokens

OOV Rate

0.500%

Vocabulary Coverage

99.500%

Step by step

Vocabulary size
unique tokens
= 30,000 tokens
OOV rate
(25000 ÷ 5000000) × 100
= 0.500%
Coverage rate
100 − 0.500
= 99.500%

How it works

Vocabulary size is the count of unique tokens a tokenizer or model recognizes, and the out-of-vocabulary (OOV) rate measures how often tokens in real text fall outside that vocabulary and must be mapped to an <unk> token — computed as unknown token occurrences divided by total tokens, times 100. A high OOV rate signals that the vocabulary is too small or poorly matched to the target domain, leading to information loss during tokenization.

Formula

oov_rate = (unknown_tokens / total_tokens) × 100

unknown_tokens: Number of token occurrences not in vocabulary
total_tokens: Total token occurrences in the corpus

Frequently Asked Questions

What's a typical OOV rate for a well-trained tokenizer?

Modern subword tokenizers (BPE, WordPiece, SentencePiece) rarely produce true OOV tokens since they can decompose any word into known subword units, achieving near-0% OOV rates — high OOV rates are more common with word-level (not subword) tokenizers.

How does vocabulary size affect model size?

The embedding and output layers scale with vocabulary size, so a larger vocabulary increases the model's parameter count and memory footprint, though it may reduce sequence lengths for the same text.

Should I always maximize vocabulary size to minimize OOV?

No — larger vocabularies increase model size and can dilute training signal for rare tokens; subword tokenization is generally preferred over simply growing a word-level vocabulary indefinitely.

You might also need

Tokenizer Comparison Calculator Sequence Padding Calculator Sequence Truncation Calculator