Calculate Huber loss, a robust regression loss combining MSE and MAE behavior.
Huber loss combines the best properties of MSE and MAE: for small residuals (|a| ≤ δ), it behaves like squared error, Huber(a) = 0.5 × a²; for large residuals (|a| > δ), it switches to a linear penalty, Huber(a) = δ × (|a| − 0.5 × δ). This makes it quadratic (and smoothly differentiable) near zero for stable gradients, while remaining linear — and therefore robust to outliers — for large errors, unlike pure MSE which lets outliers dominate the loss. The delta parameter controls where this transition happens: smaller delta makes the loss behave more like MAE, larger delta makes it behave more like MSE.
Quadratic region (|a| ≤ δ)
L = 0.5 * a^2
Linear region (|a| > δ)
L = delta * (|a| - 0.5 * delta)
Delta should reflect the scale of 'normal' residuals you expect — errors smaller than delta are treated as ordinary noise (quadratic penalty), while errors larger than delta are treated as likely outliers (linear penalty); it's often tuned via cross-validation or set based on domain knowledge of acceptable error magnitude.
MSE squares every error, so a few extreme outliers can dominate the total loss and distort model training; Huber loss caps the penalty growth to linear beyond delta, making the model less sensitive to a small number of extreme residuals.
As delta → ∞, Huber loss behaves exactly like MSE everywhere (always quadratic); as delta → 0, it behaves increasingly like MAE (nearly always linear), so delta effectively interpolates between the two loss functions.