GELU Activation Calculator
Calculate the GELU (Gaussian Error Linear Unit) activation output using the standard tanh approximation.
Inputs
GELU(x)
0.841192
tanh(inner term)
0.682384
Step by step
Inner term: sqrt(2/π) × (x + 0.044715 × x³)
sqrt(2/π) × (1 + 0.044715 × 1³)
= 0.833562
tanh(inner term)
tanh(0.833562)
= 0.682384
GELU(x) = 0.5 × x × (1 + tanh(inner))
0.5 × 1 × (1 + 0.682384)
= 0.841192
How it works
GELU weights its input by the probability that a standard Gaussian random variable is less than that input, giving a smooth, non-monotonic curve rather than ReLU's sharp corner at zero. Since the exact Gaussian CDF is expensive to compute, the standard approximation used in practice (and implemented here) is GELU(x) ≈ 0.5·x·(1 + tanh(√(2/π)·(x + 0.044715·x³))). GELU is the default activation function in BERT, GPT-2/3/4, and most modern transformer architectures, since its smoothness (unlike ReLU's kink) tends to improve optimization in very deep networks.
Formula
GELU(x) = 0.5 × x × (1 + tanh(sqrt(2/pi) × (x + 0.044715 × x^3)))
- x
- Input value
- tanh
- Hyperbolic tangent function
- pi
- Pi (≈3.14159)
Frequently Asked Questions
How is GELU different from ReLU?
ReLU is a sharp, piecewise-linear function with a hard corner at x=0, while GELU is smooth and slightly non-monotonic — it can output small negative values for inputs just below 0, weighting each input by how likely it is to be 'kept' under a Gaussian distribution, rather than a hard cutoff.
Why do transformers like BERT and GPT use GELU instead of ReLU?
GELU's smoothness (having continuous derivatives everywhere) tends to produce better gradient flow and empirically improved performance in very deep transformer stacks compared to ReLU's non-smooth kink at zero, which is why it became the de facto standard for transformer feed-forward blocks.
Why use a tanh approximation instead of the exact GELU formula?
The exact GELU involves the Gaussian error function (erf), which has no simple closed-form and is relatively expensive to compute on GPUs at scale; the tanh-based polynomial approximation used here matches the exact function to within about 0.1% while being much faster to evaluate.