Skip to content
ENExplore Calculators

Throughput (Tokens Per Second) Calculator

Work out throughput (tokens per second) instantly with clear inputs, formula shown and shareable results.

About Throughput (Tokens Per Second) Calculator

Serving throughput is per-stream generation speed multiplied by the number of streams a batch can hold, multiplied again by replica count. Continuous batching is what makes this work: while one sequence waits on its next token, others are decoded in the same kernel launch, so aggregate throughput scales far better than single-stream speed suggests. Dividing by tokens per request converts it into completed requests.

Formula

Serving throughput

total tok/s = concurrency x per-stream tok/s x GPUs; requests/min = total tok/s x 60 / output tokens per request

Worked example

With the default values below, Throughput (Tokens Per Second) Calculator returns Aggregate output throughput of 1,344.0. Change any input above and every figure here is recalculated in your browser.

Inputs used

Concurrent streamsminimum 1
16
Tokens per second per streamminimum 0.1 tok/s
42 tok/s
Output tokens per requestminimum 1
512
GPUs servingminimum 1
2

Results

Aggregate output throughput
1,344.0 tok/s
Tokens per minute
80,640
Completed requests per minute
157.5
Throughput per GPU
672.0 tok/s

Step by step

  1. 1Values used
  2. 2Serving throughput
  3. 3Aggregate output throughput= 1,344.0 tok/s
  4. 4Tokens per minute= 80,640
  5. 5Completed requests per minute= 157.5
  6. 6Throughput per GPU= 672.0 tok/s

Assumptions and limits

  • The result is derived only from the 4 inputs listed above. Anything not among them is not part of the calculation.
  • In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
  • Figures are rounded for display to 1 decimal place, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.

How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.

Frequently Asked Questions

Why does per-stream speed fall as concurrency rises?

Decoding is memory-bandwidth bound, and a larger batch shares that bandwidth. Aggregate throughput still rises, but each individual user sees slower streaming.

What limits concurrency?

KV cache memory. Each active sequence reserves cache proportional to its context length, so long contexts cut the number of streams that fit.

Related Calculators

Throughput (Tokens Per Second) Calculator | Calcrivo