Leaky ReLU Calculator
Compute the Leaky ReLU activation output with a configurable negative slope.
About Leaky ReLU Calculator
Leaky ReLU addresses the 'dying ReLU' problem by allowing a small, non-zero gradient for negative inputs instead of a hard zero: LeakyReLU(x) = x for x > 0, and α·x for x ≤ 0, where α (default 0.01) is a small fixed slope. Its derivative is 1 for positive inputs and α for negative inputs — never exactly 0 — so neurons that receive negative inputs can still receive small gradient updates and potentially recover during training, rather than becoming permanently inactive as with standard ReLU.
Formula
LeakyReLU(x) = x if x > 0, else alpha × x
- x
- Input value
- alpha
- Negative slope (default 0.01)
Worked example
With the default values below, Leaky ReLU Calculator returns LeakyReLU(x) of -0.020000. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Input (x)accepts -1,000–1,000
- -2
- Alpha (negative slope)accepts 0–1
- 0.01
Results
- LeakyReLU(x)
- -0.020000
- LeakyReLU'(x)
- 0.0100
Step by step
- 1LeakyReLU(x) = α × x (negative branch)= -0.020000
- 2Derivative: 1 if x > 0, else α= 0.0100
Assumptions and limits
- The result is derived only from the 2 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 6 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
How does Leaky ReLU solve the dying ReLU problem?
Because the derivative for negative inputs is α (a small positive number) rather than exactly 0, neurons with negative pre-activations still receive a (small) gradient signal during backpropagation, giving them a chance to adjust their weights and become active again instead of getting permanently stuck.
What's a typical value for alpha?
0.01 is the most common default (matching the original Leaky ReLU paper), though some architectures use larger values like 0.1 or 0.2, or make alpha a learnable parameter — a variant known as Parametric ReLU (PReLU).
Is Leaky ReLU always better than standard ReLU?
Not necessarily — in practice the difference in final model performance is often small, and standard ReLU's sparsity (exact zeros for negative inputs) can sometimes act as a useful implicit regularizer; Leaky ReLU is typically tried as one of several activation options during hyperparameter search rather than as an automatic upgrade.