Calculate the maximum request throughput an API gateway instance can sustain.
With a fixed backend connection pool, each connection can serve only one request at a time, so the gateway's maximum throughput is bound by how many connections it has and how long each request occupies one. Formula: max_rps = connection_pool_size / (backend_latency_ms / 1000), reduced by the gateway's own processing overhead (auth, routing, transformation) to get an effective, realistic ceiling.
Effective max throughput
effective_max_rps = (connection_pool_size / (backend_latency_ms / 1000)) × (1 - overhead_percent / 100)
Increase the connection pool size (if the backend can handle more concurrent connections) or reduce backend latency (caching, query optimization) — both directly raise the theoretical ceiling in the formula.
Check for connection pool exhaustion/queueing, keep-alive settings, DNS resolution overhead, or backend latency spikes under load — real-world throughput is often below the theoretical figure due to tail latency, not just averages.
API gateways add real processing time for authentication, rate limiting, request/response transformation and logging — this can be a meaningful percentage of the total request time, especially for fast backends where gateway overhead is a bigger share of total latency.
No — this model assumes one request occupies one pooled connection for its full duration (like a typical HTTP/1.1 keep-alive pool). Multiplexed protocols (HTTP/2, gRPC) can serve multiple concurrent requests per connection, which would require a different concurrency model.