Skip to content
ENExplore Calculators

A B Test For Models Calculator

Work out a b test for models instantly with clear inputs, formula shown and shareable results.

About A B Test For Models Calculator

Comparing two models on a conversion metric is a two-proportion test. Required sample per arm scales with the variance of the metric and inversely with the square of the effect size, so halving the detectable effect quadruples the sample. That relationship is why tiny model improvements are effectively unmeasurable at modest traffic, and why the minimum detectable effect must be agreed before launch.

Formula

Two-proportion sample size

n per arm = (z_alpha sqrt(2 p(1-p)) + z_beta sqrt(p1(1-p1) + p2(1-p2)))^2 / delta^2

delta
Absolute difference to detect
z_alpha
1.96 for two-sided 95%
z_beta
0.84 for 80% power

Worked example

With the default values below, A B Test For Models Calculator returns Visitors needed per arm of 7,761. Change any input above and every figure here is recalculated in your browser.

Inputs used

Baseline conversion rateaccepts 0.01–99.9 %
12 %
Minimum detectable effect (absolute)accepts 0.01–50 %
1.5 %
Daily visitors across both armsminimum 1
40,000
Statistical poweraccepts 50–99 %
80 %

Results

Visitors needed per arm
7,761
Total visitors needed
15,521
Days to reach the sample
0.4 days
Relative lift being tested for
12.50%

Step by step

  1. 1Values used
  2. 2Two-proportion sample size
  3. 3Visitors needed per arm= 7,761
  4. 4Total visitors needed= 15,521
  5. 5Days to reach the sample= 0.4 days
  6. 6Relative lift being tested for= 12.50

Assumptions and limits

  • The result is derived only from the 4 inputs listed above. Anything not among them is not part of the calculation.
  • In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
  • Figures are rounded for display to 2 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.

How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.

Frequently Asked Questions

Can I stop the test early if it looks significant?

Not with a fixed-horizon test; peeking inflates the false-positive rate badly. Use a sequential or Bayesian design if you need to monitor continuously.

Should I split traffic evenly?

Even splits minimise the total sample needed. Uneven splits are for risk management, and they increase the required duration.

Related Calculators

A B Test For Models Calculator | Calcrivo