Flaky Test Rate Calculator

Work out flaky test rate instantly with clear inputs, formula shown and shareable results.

About Flaky Test Rate Calculator

A flaky test is one that fails and then passes on retry without any code change. The damaging metric is the share of all failures that are flaky: once it passes roughly half, developers stop reading failures and start reflexively retrying, which means genuine regressions reach production. The wasted compute is real but secondary to that loss of trust.

Formula

Flakiness

flaky rate = flaky failures / runs; false failure share = flaky failures / total failures; wasted time = flaky failures x rerun duration

Worked example

With the default values below, Flaky Test Rate Calculator returns Flaky failure rate of 10.17%. Change any input above and every figure here is recalculated in your browser.

Inputs used

Pipeline runsminimum 1: 1,200
Runs that failedminimum 0: 168
Failures that passed on retryminimum 0: 122
Time cost of a rerunminimum 0.1 min: 14 min

Results

Flaky failure rate: 10.17%
Share of failures that were flaky: 72.6%
Genuine failures: 46
Compute hours wasted on reruns: 28.5 h

Step by step

1Values used
2Flakiness
3Flaky failure rate= 10.17
4Share of failures that were flaky= 72.6
5Genuine failures= 46
6Compute hours wasted on reruns= 28.5 h

Assumptions and limits

The result is derived only from the 4 inputs listed above. Anything not among them is not part of the calculation.
In particular, this calculator does not take inflation, tax or compounding frequency into account.
Figures are rounded for display to 2 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.

How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.

Frequently Asked Questions

Should flaky tests be quarantined or fixed?

Quarantine immediately to restore signal, then fix on a deadline. Leaving them in the blocking suite trains the team to ignore red builds.

What causes most flakiness?

Timing assumptions and shared mutable state — fixed sleeps instead of waiting for a condition, test order dependence, and unreset databases or caches between tests.