Work out relu activation output instantly with clear inputs, formula shown and shareable results.
ReLU passes positive values unchanged and clamps negatives to zero, which keeps gradients at exactly 1 in the active region and avoids the vanishing-gradient problem of sigmoid and tanh. Its weakness is that a unit stuck at negative pre-activation has zero gradient forever. Leaky ReLU keeps a small slope alpha there, and ELU uses a smooth exponential branch that also centres activations nearer zero.
Activation definitions
ReLU: f(x) = max(0, x); Leaky: f(x) = x if x > 0 else alpha x; ELU: f(x) = x if x > 0 else alpha(e^x - 1)
A large negative bias or an aggressive learning rate can push a unit permanently into the negative region. Since the gradient there is exactly zero, no update can revive it.
Plain ReLU is still a strong default for convolutional networks. Transformers mostly use GELU or SwiGLU, which are smooth and slightly better empirically at the same cost.