Estimate how many tokens a piece of text will consume in an LLM prompt.
LLM tokenizers (like OpenAI's tiktoken or Anthropic's tokenizer) split text into subword units called tokens. For English text, a common rule of thumb is about 4 characters per token, or roughly 0.75 words per token. This calculator blends both heuristics to give a robust estimate without needing to run the actual tokenizer. Actual token counts vary by model and language — non-English text and code typically use more tokens per character.
tokens = (char_count / chars_per_token + word_count / 0.75) / 2
It's a heuristic approximation, typically within 10-15% of the actual token count for English prose. For exact counts, use the model provider's official tokenizer (e.g. tiktoken for OpenAI models).
Tokenizers split on subword units, not whole words. Common words are often a single token, while rare words, punctuation, and whitespace can each consume additional tokens.
Non-English languages, especially those with non-Latin scripts, often use more tokens per character than English. This calculator's default ratio is tuned for English.
Yes. Code tends to have more tokens per character than natural language because of symbols, indentation, and variable names, so real token counts for code will typically be higher than this estimate.