P-value Calculator
Calculate the p-value for a hypothesis test comparing model or experiment results.
Inputs
Standardized test statistic, e.g. from comparing two model accuracies.
P-value
1.964271
Result
Not statistically significant at α=0.05
Step by step
One-tailed p: 1 − Φ(|z|)
1 − Φ(2.1000)
= 0.982136
Two-tailed p: 2 × one-tailed p
2 × 0.982136
= 1.964271
Compare to α = 0.05
1.964271 ≥ 0.05
= Fail to reject H0
How it works
The p-value is the probability of observing a test statistic at least as extreme as the one computed, assuming the null hypothesis is true. From a z-score: for a two-tailed test, p = 2 × (1 − Φ(|z|)), accounting for extreme values in either direction; for a one-tailed test, p = 1 − Φ(|z|), considering only one direction. A small p-value (typically below the chosen significance level α, often 0.05) is taken as evidence against the null hypothesis. In ML, this is commonly used to test whether an observed difference between two models' metrics (e.g. in an A/B test or model comparison) is statistically significant or could plausibly be due to chance.
Formulas
Two-tailed p-value
p = 2 × (1 - Phi(|z|))
- z
- Standardized test statistic (z-score)
- Phi
- Standard normal cumulative distribution function
One-tailed p-value
p = 1 - Phi(|z|)
- z
- Standardized test statistic (z-score)
- Phi
- Standard normal cumulative distribution function
Frequently Asked Questions
What does a p-value of 0.03 actually mean?
It means that, assuming the null hypothesis (e.g. 'no real difference between the two models') is true, there's only a 3% chance of observing a difference this large or larger purely by chance — it does not mean there's a 3% chance the null hypothesis is true, which is a very common misinterpretation.
Why choose two-tailed vs. one-tailed?
Use a two-tailed test when you care about a difference in either direction (e.g. 'is model A different from model B, better or worse'); use a one-tailed test only when you have a specific directional hypothesis decided in advance (e.g. 'is model A better than model B') and are willing to ignore evidence in the opposite direction.
Why is 0.05 the common significance threshold?
It's a widely adopted convention (not a universal law) originating from early 20th-century statistics — meaning a 5% chance of a false positive (rejecting a true null hypothesis) is considered acceptable; more rigorous fields or high-stakes decisions sometimes use stricter thresholds like 0.01.
Does a significant p-value guarantee a practically important result?
No — with a large enough sample size, even a tiny, practically meaningless difference can produce a statistically significant p-value; always consider the effect size (the actual magnitude of the difference) alongside the p-value.