Token Calculator
Estimate how many tokens a piece of text will consume in an LLM prompt.
About Token Calculator
LLM tokenizers (like OpenAI's tiktoken or Anthropic's tokenizer) split text into subword units called tokens. For English text, a common rule of thumb is about 4 characters per token, or roughly 0.75 words per token. This calculator blends both heuristics to give a robust estimate without needing to run the actual tokenizer. Actual token counts vary by model and language — non-English text and code typically use more tokens per character.
Formula
tokens = (char_count / chars_per_token + word_count / 0.75) / 2
- char_count
- Total character count of the text
- chars_per_token
- Average characters per token (default 4 for English BPE)
- word_count
- Total word count of the text
Worked example
With the default values below, Token Calculator returns Estimated Tokens of 27. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Text
- The quick brown fox jumps over the lazy dog. Large language models split text into tokens before processing it.
- Characters per Tokenaccepts 1–10 chars/token
- 4 chars/token
Results
- Estimated Tokens
- 27 tokens
- Word Count
- 19 words
- Character Count
- 111 chars
- Words-to-Token Ratio
- 0.72
Step by step
- 1Character-based estimate: characters ÷ chars-per-token= 27.8
- 2Word-based estimate: words ÷ 0.75= 25.3
- 3Blended estimate (average of both methods)= 27
Assumptions and limits
- The result is derived only from the 2 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 2 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
How accurate is this estimate?
It's a heuristic approximation, typically within 10-15% of the actual token count for English prose. For exact counts, use the model provider's official tokenizer (e.g. tiktoken for OpenAI models).
Why do tokens differ from words?
Tokenizers split on subword units, not whole words. Common words are often a single token, while rare words, punctuation, and whitespace can each consume additional tokens.
Does this work for non-English text?
Non-English languages, especially those with non-Latin scripts, often use more tokens per character than English. This calculator's default ratio is tuned for English.
Do code snippets tokenize differently?
Yes. Code tends to have more tokens per character than natural language because of symbols, indentation, and variable names, so real token counts for code will typically be higher than this estimate.