Tokens/sec Calculator
Calculate LLM throughput in tokens generated or processed per second.
Inputs
Concurrent sequences being decoded; throughput scales roughly linearly while memory-bandwidth bound.
Tokens/Second (Single Sequence)
145.64tok/s
Aggregate Throughput (Batched)
145.64tok/s
Latency per Token
6.87ms
Step by step
Weight bytes streamed per token: params (B) × bytes/param
7B × 2 bytes
= 14.00 GB
Tokens/sec (single sequence): memory bandwidth ÷ weight bytes per token
2039 GB/s ÷ 14.00 GB
= 145.64 tok/s
Batched throughput: single-sequence tokens/sec × batch size
145.64 × 1
= 145.64 tok/s
How it works
Autoregressive LLM decoding generates one token at a time, and for each token the GPU must stream the entire model's weights from VRAM into compute units — this makes single-sequence decoding memory-bandwidth bound rather than compute bound. Throughput is approximated as tokens/sec ≈ memory_bandwidth_GB_s / (params_billions × bytes_per_param): a smaller model, faster memory, or lower-bit quantization all directly increase tokens/sec. Batching multiple sequences together amortizes this weight-streaming cost across more tokens per pass, which is why batched throughput scales up substantially even though single-sequence latency does not improve.
Formula
tokens_per_sec = memory_bandwidth_GBs / (params_billions * bytes_per_param)
- BW
- GPU memory bandwidth in GB/s
- P_B
- Model parameters in billions
- B_p
- Bytes per parameter (2 for FP16, 1 for INT8, 0.5 for INT4)
Frequently Asked Questions
Why is decoding memory-bandwidth bound rather than compute bound?
Generating a single token requires one full pass through all model weights, but that pass does relatively little arithmetic per weight loaded — so the GPU spends most of its time waiting on data movement from memory rather than being limited by its FLOPs throughput.
Why does quantization speed up inference?
Lower-precision formats like INT8 or INT4 store each parameter in fewer bytes, meaning less data has to move across the memory bus per token generated, which directly increases tokens/sec since decoding is bandwidth-bound.
Does increasing batch size increase single-sequence speed?
No — individual sequence latency stays roughly the same because each token still requires a full weight pass, but total system throughput scales because you're now generating tokens for multiple sequences per pass, amortizing the bandwidth cost.
How accurate is this compared to real-world serving frameworks?
This is a theoretical upper-bound estimate; real deployments (vLLM, TensorRT-LLM, etc.) typically achieve 40-70% of this bound due to kernel launch overhead, attention computation, and imperfect batching, so treat this as a best-case ceiling.