Work out throughput vs latency tradeoff instantly with clear inputs, formula shown and shareable results.
Throughput and latency trade off through utilisation: queueing theory allows utilisation of roughly wait/(service + wait), so a latency target only slightly above service time forces low utilisation and therefore low usable throughput. Batching raises maximum throughput by amortising fixed cost per item but increases per-request latency, which is exactly the tension the two numbers here expose.
Latency-bounded throughput
allowed wait = target - service time; permitted utilisation = wait / (service + wait); sustainable throughput = max throughput x utilisation
Because full utilisation means a permanently non-empty queue. Latency is service time plus queue time, and queue time is what utilisation buys you.
For asynchronous or bulk pipelines where per-item latency does not matter. For interactive requests, small batches with a short flush timeout are the usual compromise.