Calculate the p-value for a hypothesis test comparing model or experiment results.
The p-value is the probability of observing a test statistic at least as extreme as the one computed, assuming the null hypothesis is true. From a z-score: for a two-tailed test, p = 2 × (1 − Φ(|z|)), accounting for extreme values in either direction; for a one-tailed test, p = 1 − Φ(|z|), considering only one direction. A small p-value (typically below the chosen significance level α, often 0.05) is taken as evidence against the null hypothesis. In ML, this is commonly used to test whether an observed difference between two models' metrics (e.g. in an A/B test or model comparison) is statistically significant or could plausibly be due to chance.
Two-tailed p-value
p = 2 × (1 - Phi(|z|))
One-tailed p-value
p = 1 - Phi(|z|)
It means that, assuming the null hypothesis (e.g. 'no real difference between the two models') is true, there's only a 3% chance of observing a difference this large or larger purely by chance — it does not mean there's a 3% chance the null hypothesis is true, which is a very common misinterpretation.
Use a two-tailed test when you care about a difference in either direction (e.g. 'is model A different from model B, better or worse'); use a one-tailed test only when you have a specific directional hypothesis decided in advance (e.g. 'is model A better than model B') and are willing to ignore evidence in the opposite direction.
It's a widely adopted convention (not a universal law) originating from early 20th-century statistics — meaning a 5% chance of a false positive (rejecting a true null hypothesis) is considered acceptable; more rigorous fields or high-stakes decisions sometimes use stricter thresholds like 0.01.
No — with a large enough sample size, even a tiny, practically meaningless difference can produce a statistically significant p-value; always consider the effect size (the actual magnitude of the difference) alongside the p-value.