Tokens/sec Calculator
Calculate LLM throughput in tokens generated or processed per second.
About Tokens/sec Calculator
Autoregressive LLM decoding generates one token at a time, and for each token the GPU must stream the entire model's weights from VRAM into compute units — this makes single-sequence decoding memory-bandwidth bound rather than compute bound. Throughput is approximated as tokens/sec ≈ memory_bandwidth_GB_s / (params_billions × bytes_per_param): a smaller model, faster memory, or lower-bit quantization all directly increase tokens/sec. Batching multiple sequences together amortizes this weight-streaming cost across more tokens per pass, which is why batched throughput scales up substantially even though single-sequence latency does not improve.
Formula
tokens_per_sec = memory_bandwidth_GBs / (params_billions * bytes_per_param)
- BW
- GPU memory bandwidth in GB/s
- P_B
- Model parameters in billions
- B_p
- Bytes per parameter (2 for FP16, 1 for INT8, 0.5 for INT4)
Worked example
With the default values below, Tokens/sec Calculator returns Tokens/Second (Single Sequence) of 145.64. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Model Parametersaccepts 0.01–2,000 billion
- 7 billion
- Hardwareone of NVIDIA A100 (2039 GB/s), NVIDIA H100 (3350 GB/s), NVIDIA RTX 4090 (1008 GB/s)
- NVIDIA A100 (2039 GB/s)
- Quantizationone of FP16 / BF16 (2 bytes), INT8 (1 byte), INT4 (0.5 bytes)
- FP16 / BF16 (2 bytes)
- Batch Sizeaccepts 1–10,000 sequences
- 1 sequences
Results
- Tokens/Second (Single Sequence)
- 145.64 tok/s
- Aggregate Throughput (Batched)
- 145.64 tok/s
- Latency per Token
- 6.87 ms
Step by step
- 1Weight bytes streamed per token: params (B) × bytes/param= 14.00 GB
- 2Tokens/sec (single sequence): memory bandwidth ÷ weight bytes per token= 145.64 tok/s
- 3Batched throughput: single-sequence tokens/sec × batch size= 145.64 tok/s
Assumptions and limits
- The result is derived only from the 4 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 2 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
Why is decoding memory-bandwidth bound rather than compute bound?
Generating a single token requires one full pass through all model weights, but that pass does relatively little arithmetic per weight loaded — so the GPU spends most of its time waiting on data movement from memory rather than being limited by its FLOPs throughput.
Why does quantization speed up inference?
Lower-precision formats like INT8 or INT4 store each parameter in fewer bytes, meaning less data has to move across the memory bus per token generated, which directly increases tokens/sec since decoding is bandwidth-bound.
Does increasing batch size increase single-sequence speed?
No — individual sequence latency stays roughly the same because each token still requires a full weight pass, but total system throughput scales because you're now generating tokens for multiple sequences per pass, amortizing the bandwidth cost.
How accurate is this compared to real-world serving frameworks?
This is a theoretical upper-bound estimate; real deployments (vLLM, TensorRT-LLM, etc.) typically achieve 40-70% of this bound due to kernel launch overhead, attention computation, and imperfect batching, so treat this as a best-case ceiling.