Compute the Leaky ReLU activation output with a configurable negative slope.
Leaky ReLU addresses the 'dying ReLU' problem by allowing a small, non-zero gradient for negative inputs instead of a hard zero: LeakyReLU(x) = x for x > 0, and α·x for x ≤ 0, where α (default 0.01) is a small fixed slope. Its derivative is 1 for positive inputs and α for negative inputs — never exactly 0 — so neurons that receive negative inputs can still receive small gradient updates and potentially recover during training, rather than becoming permanently inactive as with standard ReLU.
LeakyReLU(x) = x if x > 0, else alpha × x
Because the derivative for negative inputs is α (a small positive number) rather than exactly 0, neurons with negative pre-activations still receive a (small) gradient signal during backpropagation, giving them a chance to adjust their weights and become active again instead of getting permanently stuck.
0.01 is the most common default (matching the original Leaky ReLU paper), though some architectures use larger values like 0.1 or 0.2, or make alpha a learnable parameter — a variant known as Parametric ReLU (PReLU).
Not necessarily — in practice the difference in final model performance is often small, and standard ReLU's sparsity (exact zeros for negative inputs) can sometimes act as a useful implicit regularizer; Leaky ReLU is typically tried as one of several activation options during hyperparameter search rather than as an automatic upgrade.