Estimate the vocabulary size needed for a tokenizer given corpus statistics.
Vocabulary size is the count of unique tokens a tokenizer or model recognizes, and the out-of-vocabulary (OOV) rate measures how often tokens in real text fall outside that vocabulary and must be mapped to an <unk> token — computed as unknown token occurrences divided by total tokens, times 100. A high OOV rate signals that the vocabulary is too small or poorly matched to the target domain, leading to information loss during tokenization.
oov_rate = (unknown_tokens / total_tokens) × 100
Modern subword tokenizers (BPE, WordPiece, SentencePiece) rarely produce true OOV tokens since they can decompose any word into known subword units, achieving near-0% OOV rates — high OOV rates are more common with word-level (not subword) tokenizers.
The embedding and output layers scale with vocabulary size, so a larger vocabulary increases the model's parameter count and memory footprint, though it may reduce sequence lengths for the same text.
No — larger vocabularies increase model size and can dilute training signal for rare tokens; subword tokenization is generally preferred over simply growing a word-level vocabulary indefinitely.