Throughput vs Latency Tradeoff Calculator

Work out throughput vs latency tradeoff instantly with clear inputs, formula shown and shareable results.

About Throughput vs Latency Tradeoff Calculator

Throughput and latency trade off through utilisation: queueing theory allows utilisation of roughly wait/(service + wait), so a latency target only slightly above service time forces low utilisation and therefore low usable throughput. Batching raises maximum throughput by amortising fixed cost per item but increases per-request latency, which is exactly the tension the two numbers here expose.

Formula

Latency-bounded throughput

allowed wait = target - service time; permitted utilisation = wait / (service + wait); sustainable throughput = max throughput x utilisation

Worked example

With the default values below, Throughput vs Latency Tradeoff Calculator returns Throughput achievable at the latency target of 533. Change any input above and every figure here is recalculated in your browser.

Inputs used

Service time per requestminimum 0.01 ms: 20 ms
Parallel servers or threadsminimum 1: 16
Latency targetminimum 0.01 ms: 60 ms
Batch sizeaccepts 1–1,000: 1

Results

Throughput achievable at the latency target: 533 req/s
Maximum throughput ignoring latency: 800 req/s
Utilisation the target allows: 0.667
Effective service time with batching: 20.000 ms

Step by step

1Values used
2Latency-bounded throughput
3Throughput achievable at the latency target= 533 req/s
4Maximum throughput ignoring latency= 800 req/s
5Utilisation the target allows= 0.667
6Effective service time with batching= 20.000 ms

Assumptions and limits

The result is derived only from the 4 inputs listed above. Anything not among them is not part of the calculation.
In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
Figures are rounded for display to 3 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.

How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.

Frequently Asked Questions

Why can I not run at full throughput and low latency?

Because full utilisation means a permanently non-empty queue. Latency is service time plus queue time, and queue time is what utilisation buys you.

When is batching worth the latency cost?

For asynchronous or bulk pipelines where per-item latency does not matter. For interactive requests, small batches with a short flush timeout are the usual compromise.