Calculate optimal thread pool size for a service based on request rate and latency.
The classic thread pool sizing formula — threads = cores × (1 + wait_time/service_time) — captures a simple insight: while a thread is blocked waiting (on I/O, a lock, or a network call), its CPU core sits idle unless another thread is available to run, so the higher the ratio of wait time to actual CPU service time, the more threads are needed to keep cores busy. Purely CPU-bound work (near-zero wait) needs a pool close to core count, since extra threads there mostly add context-switching overhead without more useful parallelism; heavily I/O-bound work justifies a much larger pool (or, at scale, switching to an async/event-driven model that doesn't need one OS thread per in-flight task at all).
Optimal thread pool size
optimal_threads = cores × (1 + wait_time / service_time) × target_utilization%
Beyond the point where cores are kept fully busy, additional threads mostly add context-switching overhead (cache thrashing, scheduler bookkeeping) and increased memory usage (each thread needs its own stack) without contributing more useful parallel work — for CPU-bound workloads especially, pool sizes far beyond core count often reduce throughput rather than increase it.
Profile a representative task and measure time spent blocked (in syscalls like read()/recv(), or waiting on a lock/database query) versus time spent actively executing CPU instructions — tools like `strace -T` (per-syscall timing) or application-level tracing/APM tools can break this down directly.
Not directly — this sizing model assumes a traditional blocking-thread-per-task model where a blocked thread consumes a pool slot doing nothing useful. Async/event-driven architectures (Node.js, async Rust/Python) sidestep the need for large thread pools by handling many in-flight I/O operations on far fewer threads, since a single thread can juggle many non-blocking operations concurrently.