Compute the Exponential Linear Unit activation function for a given input.
ELU behaves like the identity for positive inputs but smoothly saturates toward −α for large negative inputs instead of flattening to a fixed slope: ELU(x) = x for x > 0, and α·(e^x − 1) for x ≤ 0. Unlike ReLU and Leaky ReLU, ELU is smooth (continuously differentiable) at x=0, and its negative saturation value pushes mean activations closer to zero across a layer, which can speed up training convergence compared to ReLU-family functions that only allow non-negative (or unboundedly negative, for Leaky ReLU) outputs.
ELU(x) = x if x > 0, else alpha × (e^x - 1)
The exponential term e^x approaches 0 as x becomes very negative, so α×(e^x − 1) approaches −α — this bounded negative saturation (rather than an unbounded linear decrease) helps push a layer's mean pre-activation closer to zero, which some research suggests improves training dynamics.
Yes — unlike ReLU and Leaky ReLU which have a sharp corner at x=0, ELU's positive and negative branches meet with matching slopes at x=0 when derivative continuity is checked, making it a smooth (C¹ continuous) function throughout.
1.0 is the standard default from the original ELU paper and is used in the vast majority of implementations; larger alpha values increase how negative the saturation point can go, but 1.0 works well across most architectures without additional tuning.