A/B Test Significance Calculator
Test whether the difference between two conversion rates is statistically significant.
About A/B Test Significance Calculator
The pooled standard error assumes both variants share one true conversion rate, which is the null hypothesis being tested. The z-score measures how many standard errors apart the observed rates are, and the p-value converts that into a probability. Calling a winner before reaching significance is the most common testing error, and stopping the moment a result looks good inflates the false positive rate well above the nominal 5 per cent.
Formula
A/B Test Significance
z = (pB − pA) ÷ √(p̄(1 − p̄)(1/nA + 1/nB)); p-value from the normal distribution
- pooled rate
- Combined conversion rate across both variants, used for the standard error
- p-value
- Probability of seeing this difference if the variants were truly identical
Worked example
With the default values below, A/B Test Significance Calculator returns Two-Tailed p-Value of 0.00578. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Visitors in variant Aminimum 1
- 4,800
- Conversions in variant Aminimum 0
- 384
- Visitors in variant Bminimum 1
- 4,750
- Conversions in variant Bminimum 0
- 456
Results
- Two-Tailed p-Value
- 0.00578
- Verdict
- Significant at the 99 per cent confidence level
- Conversion Rate of A
- 8.000%
- Conversion Rate of B
- 9.600%
- Relative Lift of B Over A
- 20.000%
Step by step
- 1Values used
- 2A/B Test Significance
- 3Two-Tailed p-Value= 0.00578
- 4Verdict= Significant at the 99 per cent confidence level
- 5Conversion Rate of A= 8.000
- 6Conversion Rate of B= 9.600
- 7Relative Lift of B Over A= 20.000
Assumptions and limits
- The result is derived only from the 4 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 5 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
How is A/B Test Significance calculated?
z = (pB − pA) ÷ √(p̄(1 − p̄)(1/nA + 1/nB)); p-value from the normal distribution The pooled standard error assumes both variants share one true conversion rate, which is the null hypothesis being tested. The z-score measures how many standard errors apart the observed rates are, and the p-value converts that into a probability.
Why does A/B Test Significance matter?
Calling a winner before reaching significance is the most common testing error, and stopping the moment a result looks good inflates the false positive rate well above the nominal 5 per cent.
What values do I need to enter?
This calculator takes 4 inputs: Visitors in variant A, Conversions in variant A, Visitors in variant B, Conversions in variant B. The pre-filled defaults are a realistic starting point — replace them with figures from your own environment for a result you can act on.