ReLU Activation Calculator
Calculate the ReLU (Rectified Linear Unit) activation output and its derivative for a given input.
Inputs
ReLU(x)
2.000000
ReLU'(x)
1
Step by step
ReLU(x) = max(0, x)
max(0, 2)
= 2.000000
Derivative: 1 if x > 0, else 0
2 > 0
= 1
How it works
ReLU is the most widely used activation function in modern deep learning: ReLU(x) = max(0, x), passing positive inputs through unchanged and zeroing out negative inputs entirely. Its derivative is simply 1 for x > 0 and 0 for x < 0 (undefined exactly at x=0, conventionally taken as 0 or 1), making it extremely cheap to compute and avoiding the vanishing gradient problem for positive inputs. The tradeoff is the 'dying ReLU' problem: once a neuron's input becomes negative, its gradient is exactly 0, so it can get permanently 'stuck' and never update again — this motivated variants like Leaky ReLU and ELU.
Formula
ReLU(x) = max(0, x)
- x
- Input value
Frequently Asked Questions
What is the 'dying ReLU' problem?
If a neuron's weighted input becomes negative and stays negative across training (e.g. due to a large negative bias update), ReLU outputs exactly 0 and its gradient is also exactly 0, meaning that neuron receives no further weight updates and effectively stops contributing to the network permanently.
Why is ReLU so computationally cheap?
It requires only a single comparison against zero (max(0, x)), with no exponentials, divisions, or trigonometric functions — this simplicity is a major reason it became the default choice for hidden layers in large-scale deep learning, where activation functions are computed billions of times.
What is ReLU's derivative exactly at x = 0?
Mathematically the derivative is undefined at exactly x=0 since the function has a sharp corner there, but in practice frameworks conventionally define it as either 0 or 1 at that single point — the choice has negligible practical impact since inputs are rarely exactly 0.