Work out throughput (tokens per second) instantly with clear inputs, formula shown and shareable results.
Serving throughput is per-stream generation speed multiplied by the number of streams a batch can hold, multiplied again by replica count. Continuous batching is what makes this work: while one sequence waits on its next token, others are decoded in the same kernel launch, so aggregate throughput scales far better than single-stream speed suggests. Dividing by tokens per request converts it into completed requests.
Serving throughput
total tok/s = concurrency x per-stream tok/s x GPUs; requests/min = total tok/s x 60 / output tokens per request
Decoding is memory-bandwidth bound, and a larger batch shares that bandwidth. Aggregate throughput still rises, but each individual user sees slower streaming.
KV cache memory. Each active sequence reserves cache proportional to its context length, so long contexts cut the number of streams that fit.