Token Calculator
Estimate how many tokens a piece of text will consume in an LLM prompt using character and word heuristics.
Inputs
Paste the text you want to estimate token usage for.
Rule of thumb for English text encoded with BPE tokenizers (GPT/Claude family).
Estimated Tokens
27tokens
Word Count
19words
Character Count
111chars
Words-to-Token Ratio
0.72
Step by step
Character-based estimate: characters ÷ chars-per-token
111 ÷ 4
= 27.8
Word-based estimate: words ÷ 0.75
19 ÷ 0.75
= 25.3
Blended estimate (average of both methods)
(27.8 + 25.3) ÷ 2
= 27
How it works
LLM tokenizers (like OpenAI's tiktoken or Anthropic's tokenizer) split text into subword units called tokens. For English text, a common rule of thumb is about 4 characters per token, or roughly 0.75 words per token. This calculator blends both heuristics to give a robust estimate without needing to run the actual tokenizer. Actual token counts vary by model and language — non-English text and code typically use more tokens per character.
Formula
tokens = (char_count / chars_per_token + word_count / 0.75) / 2
- char_count
- Total character count of the text
- chars_per_token
- Average characters per token (default 4 for English BPE)
- word_count
- Total word count of the text
Frequently Asked Questions
How accurate is this estimate?
It's a heuristic approximation, typically within 10-15% of the actual token count for English prose. For exact counts, use the model provider's official tokenizer (e.g. tiktoken for OpenAI models).
Why do tokens differ from words?
Tokenizers split on subword units, not whole words. Common words are often a single token, while rare words, punctuation, and whitespace can each consume additional tokens.
Does this work for non-English text?
Non-English languages, especially those with non-Latin scripts, often use more tokens per character than English. This calculator's default ratio is tuned for English.
Do code snippets tokenize differently?
Yes. Code tends to have more tokens per character than natural language because of symbols, indentation, and variable names, so real token counts for code will typically be higher than this estimate.