Test Execution Time Calculator
Estimate total test suite execution time from test count, parallelism, and average test duration, including added time from flaky retries.
Inputs
Total number of tests in the suite.
Number of parallel test workers/executors splitting the suite.
Average time per individual test.
Percentage of tests that fail intermittently and get retried once.
Total Test Execution Time
37.08sec
Base Execution Time
36,000.0ms
Flaky Retry Overhead Share
2.9%
Step by step
Base time = (tests ÷ parallelism) × avg duration
(2400 ÷ 8) × 120ms
= 36000.0ms
Flaky retry overhead
(2400 × 3.0% ÷ 8) × 120ms
= 1080.0ms
Total time = base + flaky overhead
36000.0ms + 1080.0ms
= 37080.0ms (37.08s)
How it works
Test suite execution time scales down roughly linearly with parallelism — splitting tests across more workers divides the base execution time — but flaky tests that fail and get retried effectively run twice, adding overhead proportional to the flaky rate. Modeling that overhead separately makes clear that flaky tests cost real pipeline time even when they eventually pass, which is often an underappreciated tax on CI throughput distinct from genuine test failures.
Formula
totalTime = (tests / parallelism × avgDuration) + (tests × flakyRate / parallelism × avgDuration)
- N
- Total tests
- t
- Average test duration in ms
- f
- Flaky test rate (as decimal)
- P
- Parallelism (workers)
- T
- Total execution time in ms
Frequently Asked Questions
Does parallelism reduce flaky test overhead too?
Yes — retries of flaky tests are typically distributed across the same parallel workers as the initial run, so the flaky overhead is also divided by the parallelism factor in this model, though in practice retry scheduling can add extra coordination delay not captured here.
What flaky test rate is considered acceptable?
Best-in-class suites keep flaky rates under 1%; rates above 3–5% meaningfully inflate CI time and erode trust in the suite, often signalling shared test state, timing assumptions, or external service dependencies that need fixing rather than just retrying around.
Is retrying flaky tests once always sufficient?
Not necessarily — some CI systems retry failed tests multiple times before marking the suite as failed; each additional retry attempt compounds the overhead this calculator estimates, so suites with aggressive multi-retry policies should scale the flaky overhead accordingly.