Calculate a confidence interval around a model metric estimate or sample mean.
A confidence interval expresses the uncertainty around a sample estimate (such as a model's average accuracy across evaluation runs): CI = mean ± z × (std / √n), where z is the critical value for the chosen confidence level (1.96 for 95%). A 95% confidence interval means that if the sampling process were repeated many times, about 95% of the resulting intervals would contain the true population value. In ML, confidence intervals are commonly reported around cross-validation metrics to communicate how much a reported accuracy or score might vary due to sampling.
CI = mean +/- z × (std / sqrt(n))
The standard error (σ/√n) shrinks as n grows, since more observations give a more precise estimate of the true mean — this is why evaluating a model on more test samples or more cross-validation folds produces a tighter, more trustworthy confidence interval.
It does not mean there's a 95% chance the true value lies in this specific interval — it means that if you repeated the sampling process many times and built an interval each time using the same method, about 95% of those intervals would contain the true population parameter.
Use the z-distribution when the sample size is large (n ≥ 30) or the population standard deviation is known; for small samples with an estimated standard deviation, a t-distribution (with n−1 degrees of freedom) gives a more accurate, slightly wider interval.
If two models' confidence intervals for a metric (e.g. accuracy) don't overlap, that's suggestive evidence one model outperforms the other; if they do overlap substantially, the observed difference may not be statistically significant — a proper hypothesis test (see the P-value Calculator) is more rigorous for this comparison.