Calculate the variance of a dataset used to measure feature spread in ML pipelines.
Variance measures how spread out a dataset is around its mean: var = Σ(x − mean)² / n for the population variance, or Σ(x − mean)² / (n−1) for the sample variance. The (n−1) divisor (Bessel's correction) makes the sample variance an unbiased estimator of the true population variance when working from a sample. In ML, variance is central to feature scaling, understanding model output stability (variance in predictions across bootstrap samples), and diagnosing overfitting via the bias-variance tradeoff.
Sample variance
s^2 = sum((x_i - mean)^2) / (n - 1)
Population variance
sigma^2 = sum((x_i - mean)^2) / n
Use sample variance (÷ n−1) when your data is a subset drawn from a larger population — the common case in ML datasets; use population variance (÷ n) only when your data represents the entire population of interest.
Dividing by n tends to underestimate the true population variance when using the sample mean (rather than the unknown true mean) in the calculation; dividing by n−1 corrects this bias, which is known as Bessel's correction.
Standard deviation is simply the square root of variance — variance is in squared units of the original data, while standard deviation is in the same units, making it easier to interpret directly.
In ML, 'variance' in this context refers to how much a model's predictions change across different training sets — high-variance models (like deep decision trees) overfit noise, while high-bias models (like linear regression on nonlinear data) underfit; the mathematical variance computed here is the building block for that broader concept.