Test whether the difference between two conversion rates is statistically significant.
The pooled standard error assumes both variants share one true conversion rate, which is the null hypothesis being tested. The z-score measures how many standard errors apart the observed rates are, and the p-value converts that into a probability. Calling a winner before reaching significance is the most common testing error, and stopping the moment a result looks good inflates the false positive rate well above the nominal 5 per cent.
A/B Test Significance
z = (pB − pA) ÷ √(p̄(1 − p̄)(1/nA + 1/nB)); p-value from the normal distribution
z = (pB − pA) ÷ √(p̄(1 − p̄)(1/nA + 1/nB)); p-value from the normal distribution The pooled standard error assumes both variants share one true conversion rate, which is the null hypothesis being tested. The z-score measures how many standard errors apart the observed rates are, and the p-value converts that into a probability.
Calling a winner before reaching significance is the most common testing error, and stopping the moment a result looks good inflates the false positive rate well above the nominal 5 per cent.
This calculator takes 4 inputs: Visitors in variant A, Conversions in variant A, Visitors in variant B, Conversions in variant B. The pre-filled defaults are a realistic starting point — replace them with figures from your own environment for a result you can act on.