Calculate the perplexity of a language model from its cross-entropy loss.
Perplexity is the standard evaluation metric for language models, derived directly from cross-entropy loss: Perplexity = e^H (when H is measured in nats, natural log) or Perplexity = 2^H (when H is measured in bits, log base 2). Intuitively, perplexity represents the model's average 'branching factor' — the effective number of equally likely next-token choices it's weighing at each step. Lower perplexity means the model assigns higher probability to the actual next tokens, indicating a better fit to the text distribution; a perplexity of 1 would mean perfect, deterministic prediction.
perplexity = e^H (nats) or 2^H (bits)
A perplexity of 20 suggests the model is, on average, as uncertain about the next token as if it were choosing uniformly at random among about 20 equally likely options — lower perplexity indicates more confident, accurate predictions.
Cross-entropy computed with natural log (nats) converts to perplexity via e^H, while cross-entropy computed with log base 2 (bits) converts via 2^H — using the wrong exponentiation base gives a numerically different (incorrect) perplexity value.
Only if they were evaluated on the same test set with the same tokenizer — perplexity values are not directly comparable across models with different vocabularies or tokenization schemes, since the 'branching factor' interpretation depends on the token granularity.