Confidence Interval Calculator
Calculate a confidence interval around a model metric estimate or sample mean.
Inputs
E.g. a model's mean accuracy across evaluation runs.
Lower Bound
0.8321
Upper Bound
0.8679
Margin of Error
0.017892
Standard Error
0.009129
Z-value Used
1.960
Step by step
Standard error: σ ÷ √n
0.05 ÷ √30
= 0.009129
Margin of error: z(95%) × SE
1.96 × 0.009129
= 0.017892
Confidence interval: mean ± margin
0.85 ± 0.017892
= [0.8321, 0.8679]
How it works
A confidence interval expresses the uncertainty around a sample estimate (such as a model's average accuracy across evaluation runs): CI = mean ± z × (std / √n), where z is the critical value for the chosen confidence level (1.96 for 95%). A 95% confidence interval means that if the sampling process were repeated many times, about 95% of the resulting intervals would contain the true population value. In ML, confidence intervals are commonly reported around cross-validation metrics to communicate how much a reported accuracy or score might vary due to sampling.
Formula
CI = mean +/- z × (std / sqrt(n))
- mean
- Sample mean
- z
- Critical value for the confidence level
- std
- Sample standard deviation
- n
- Sample size
Frequently Asked Questions
Why does a larger sample size narrow the confidence interval?
The standard error (σ/√n) shrinks as n grows, since more observations give a more precise estimate of the true mean — this is why evaluating a model on more test samples or more cross-validation folds produces a tighter, more trustworthy confidence interval.
What does '95% confidence' actually mean?
It does not mean there's a 95% chance the true value lies in this specific interval — it means that if you repeated the sampling process many times and built an interval each time using the same method, about 95% of those intervals would contain the true population parameter.
This calculator uses a z-value — when should I use a t-value instead?
Use the z-distribution when the sample size is large (n ≥ 30) or the population standard deviation is known; for small samples with an estimated standard deviation, a t-distribution (with n−1 degrees of freedom) gives a more accurate, slightly wider interval.
How is this used to compare two ML models?
If two models' confidence intervals for a metric (e.g. accuracy) don't overlap, that's suggestive evidence one model outperforms the other; if they do overlap substantially, the observed difference may not be statistically significant — a proper hypothesis test (see the P-value Calculator) is more rigorous for this comparison.