ELU Activation Calculator
Calculate the ELU (Exponential Linear Unit) activation output for a given input and alpha.
Inputs
Controls the saturation value for large negative inputs. Standard default is 1.0.
ELU(x)
-0.632121
ELU'(x)
0.367879
Step by step
ELU(x) = α × (e^x − 1) (negative branch)
1 × (e^-1 − 1)
= -0.632121
Derivative: 1 if x > 0, else ELU(x) + α
-0.632121 + 1
= 0.367879
How it works
ELU behaves like the identity for positive inputs but smoothly saturates toward −α for large negative inputs instead of flattening to a fixed slope: ELU(x) = x for x > 0, and α·(e^x − 1) for x ≤ 0. Unlike ReLU and Leaky ReLU, ELU is smooth (continuously differentiable) at x=0, and its negative saturation value pushes mean activations closer to zero across a layer, which can speed up training convergence compared to ReLU-family functions that only allow non-negative (or unboundedly negative, for Leaky ReLU) outputs.
Formula
ELU(x) = x if x > 0, else alpha × (e^x - 1)
- x
- Input value
- alpha
- Saturation magnitude for negative inputs (default 1.0)
Frequently Asked Questions
Why does ELU saturate at −α instead of continuing linearly like Leaky ReLU?
The exponential term e^x approaches 0 as x becomes very negative, so α×(e^x − 1) approaches −α — this bounded negative saturation (rather than an unbounded linear decrease) helps push a layer's mean pre-activation closer to zero, which some research suggests improves training dynamics.
Is ELU differentiable everywhere, including at x = 0?
Yes — unlike ReLU and Leaky ReLU which have a sharp corner at x=0, ELU's positive and negative branches meet with matching slopes at x=0 when derivative continuity is checked, making it a smooth (C¹ continuous) function throughout.
What's a typical value for alpha in ELU?
1.0 is the standard default from the original ELU paper and is used in the vast majority of implementations; larger alpha values increase how negative the saturation point can go, but 1.0 works well across most architectures without additional tuning.