P-value Calculator
Calculate the p-value for a hypothesis test comparing model or experiment results.
About P-value Calculator
The p-value is the probability of observing a test statistic at least as extreme as the one computed, assuming the null hypothesis is true. From a z-score: for a two-tailed test, p = 2 × (1 − Φ(|z|)), accounting for extreme values in either direction; for a one-tailed test, p = 1 − Φ(|z|), considering only one direction. A small p-value (typically below the chosen significance level α, often 0.05) is taken as evidence against the null hypothesis. In ML, this is commonly used to test whether an observed difference between two models' metrics (e.g. in an A/B test or model comparison) is statistically significant or could plausibly be due to chance.
Formulas
Two-tailed p-value
p = 2 × (1 - Phi(|z|))
- z
- Standardized test statistic (z-score)
- Phi
- Standard normal cumulative distribution function
One-tailed p-value
p = 1 - Phi(|z|)
- z
- Standardized test statistic (z-score)
- Phi
- Standard normal cumulative distribution function
Worked example
With the default values below, P-value Calculator returns P-value of 1.964271. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Z-score
- 2.1
- Test Typeone of Two-tailed, One-tailed
- Two-tailed
- Significance Level (α)one of 0.01, 0.05, 0.10
- 0.05
Results
- P-value
- 1.964271
- Result
- Not statistically significant at α=0.05
Step by step
- 1One-tailed p: 1 − Φ(|z|)= 0.982136
- 2Two-tailed p: 2 × one-tailed p= 1.964271
- 3Compare to α = 0.05= Fail to reject H0
Assumptions and limits
- The result is derived only from the 3 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 6 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
What does a p-value of 0.03 actually mean?
It means that, assuming the null hypothesis (e.g. 'no real difference between the two models') is true, there's only a 3% chance of observing a difference this large or larger purely by chance — it does not mean there's a 3% chance the null hypothesis is true, which is a very common misinterpretation.
Why choose two-tailed vs. one-tailed?
Use a two-tailed test when you care about a difference in either direction (e.g. 'is model A different from model B, better or worse'); use a one-tailed test only when you have a specific directional hypothesis decided in advance (e.g. 'is model A better than model B') and are willing to ignore evidence in the opposite direction.
Why is 0.05 the common significance threshold?
It's a widely adopted convention (not a universal law) originating from early 20th-century statistics — meaning a 5% chance of a false positive (rejecting a true null hypothesis) is considered acceptable; more rigorous fields or high-stakes decisions sometimes use stricter thresholds like 0.01.
Does a significant p-value guarantee a practically important result?
No — with a large enough sample size, even a tiny, practically meaningless difference can produce a statistically significant p-value; always consider the effect size (the actual magnitude of the difference) alongside the p-value.