ReLU Calculator
Apply the ReLU activation function to an input value or vector.
About ReLU Calculator
ReLU is the most widely used activation function in modern deep learning: ReLU(x) = max(0, x), passing positive inputs through unchanged and zeroing out negative inputs entirely. Its derivative is simply 1 for x > 0 and 0 for x < 0 (undefined exactly at x=0, conventionally taken as 0 or 1), making it extremely cheap to compute and avoiding the vanishing gradient problem for positive inputs. The tradeoff is the 'dying ReLU' problem: once a neuron's input becomes negative, its gradient is exactly 0, so it can get permanently 'stuck' and never update again — this motivated variants like Leaky ReLU and ELU.
Formula
ReLU(x) = max(0, x)
- x
- Input value
Worked example
With the default values below, ReLU Calculator returns ReLU(x) of 2.000000. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Input (x)accepts -1,000–1,000
- 2
Results
- ReLU(x)
- 2.000000
- ReLU'(x)
- 1
Step by step
- 1ReLU(x) = max(0, x)= 2.000000
- 2Derivative: 1 if x > 0, else 0= 1
Assumptions and limits
- The result is derived only from the single input listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 6 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
What is the 'dying ReLU' problem?
If a neuron's weighted input becomes negative and stays negative across training (e.g. due to a large negative bias update), ReLU outputs exactly 0 and its gradient is also exactly 0, meaning that neuron receives no further weight updates and effectively stops contributing to the network permanently.
Why is ReLU so computationally cheap?
It requires only a single comparison against zero (max(0, x)), with no exponentials, divisions, or trigonometric functions — this simplicity is a major reason it became the default choice for hidden layers in large-scale deep learning, where activation functions are computed billions of times.
What is ReLU's derivative exactly at x = 0?
Mathematically the derivative is undefined at exactly x=0 since the function has a sharp corner there, but in practice frameworks conventionally define it as either 0 or 1 at that single point — the choice has negligible practical impact since inputs are rarely exactly 0.