Leaky ReLU Calculator
Calculate the Leaky ReLU activation output, which allows a small negative slope instead of zeroing negative inputs.
Inputs
Slope applied to negative inputs. Standard default is 0.01.
LeakyReLU(x)
-0.020000
LeakyReLU'(x)
0.0100
Step by step
LeakyReLU(x) = α × x (negative branch)
0.01 × -2
= -0.020000
Derivative: 1 if x > 0, else α
-2 ≤ 0, α = 0.01
= 0.0100
How it works
Leaky ReLU addresses the 'dying ReLU' problem by allowing a small, non-zero gradient for negative inputs instead of a hard zero: LeakyReLU(x) = x for x > 0, and α·x for x ≤ 0, where α (default 0.01) is a small fixed slope. Its derivative is 1 for positive inputs and α for negative inputs — never exactly 0 — so neurons that receive negative inputs can still receive small gradient updates and potentially recover during training, rather than becoming permanently inactive as with standard ReLU.
Formula
LeakyReLU(x) = x if x > 0, else alpha × x
- x
- Input value
- alpha
- Negative slope (default 0.01)
Frequently Asked Questions
How does Leaky ReLU solve the dying ReLU problem?
Because the derivative for negative inputs is α (a small positive number) rather than exactly 0, neurons with negative pre-activations still receive a (small) gradient signal during backpropagation, giving them a chance to adjust their weights and become active again instead of getting permanently stuck.
What's a typical value for alpha?
0.01 is the most common default (matching the original Leaky ReLU paper), though some architectures use larger values like 0.1 or 0.2, or make alpha a learnable parameter — a variant known as Parametric ReLU (PReLU).
Is Leaky ReLU always better than standard ReLU?
Not necessarily — in practice the difference in final model performance is often small, and standard ReLU's sparsity (exact zeros for negative inputs) can sometimes act as a useful implicit regularizer; Leaky ReLU is typically tried as one of several activation options during hyperparameter search rather than as an automatic upgrade.