Work out concurrent request capacity instantly with clear inputs, formula shown and shareable results.
Little's law states that the average number of items in a stable system equals the arrival rate multiplied by the average time each item spends there: L = lambda x W. For an inference service that means 40 requests per second at 2.5 seconds each keeps 100 requests in flight, regardless of how the work is distributed. Dividing by the per-replica concurrency limit gives replica count directly.
Little's law
L = lambda x W; replicas = ceil(L / concurrency per replica)
No, which is what makes it so useful. It holds for any stable queueing system regardless of arrival or service-time distribution.
No. Queueing delay grows sharply above about 70 to 80 percent utilisation, so provision headroom for arrival bursts and slow requests.