Calculate LLM throughput in tokens generated or processed per second.
Autoregressive LLM decoding generates one token at a time, and for each token the GPU must stream the entire model's weights from VRAM into compute units — this makes single-sequence decoding memory-bandwidth bound rather than compute bound. Throughput is approximated as tokens/sec ≈ memory_bandwidth_GB_s / (params_billions × bytes_per_param): a smaller model, faster memory, or lower-bit quantization all directly increase tokens/sec. Batching multiple sequences together amortizes this weight-streaming cost across more tokens per pass, which is why batched throughput scales up substantially even though single-sequence latency does not improve.
tokens_per_sec = memory_bandwidth_GBs / (params_billions * bytes_per_param)
Generating a single token requires one full pass through all model weights, but that pass does relatively little arithmetic per weight loaded — so the GPU spends most of its time waiting on data movement from memory rather than being limited by its FLOPs throughput.
Lower-precision formats like INT8 or INT4 store each parameter in fewer bytes, meaning less data has to move across the memory bus per token generated, which directly increases tokens/sec since decoding is bandwidth-bound.
No — individual sequence latency stays roughly the same because each token still requires a full weight pass, but total system throughput scales because you're now generating tokens for multiple sequences per pass, amortizing the bandwidth cost.
This is a theoretical upper-bound estimate; real deployments (vLLM, TensorRT-LLM, etc.) typically achieve 40-70% of this bound due to kernel launch overhead, attention computation, and imperfect batching, so treat this as a best-case ceiling.