Compute the Gaussian Error Linear Unit activation used in transformer models.
GELU weights its input by the probability that a standard Gaussian random variable is less than that input, giving a smooth, non-monotonic curve rather than ReLU's sharp corner at zero. Since the exact Gaussian CDF is expensive to compute, the standard approximation used in practice (and implemented here) is GELU(x) ≈ 0.5·x·(1 + tanh(√(2/π)·(x + 0.044715·x³))). GELU is the default activation function in BERT, GPT-2/3/4, and most modern transformer architectures, since its smoothness (unlike ReLU's kink) tends to improve optimization in very deep networks.
GELU(x) = 0.5 × x × (1 + tanh(sqrt(2/pi) × (x + 0.044715 × x^3)))
ReLU is a sharp, piecewise-linear function with a hard corner at x=0, while GELU is smooth and slightly non-monotonic — it can output small negative values for inputs just below 0, weighting each input by how likely it is to be 'kept' under a Gaussian distribution, rather than a hard cutoff.
GELU's smoothness (having continuous derivatives everywhere) tends to produce better gradient flow and empirically improved performance in very deep transformer stacks compared to ReLU's non-smooth kink at zero, which is why it became the de facto standard for transformer feed-forward blocks.
The exact GELU involves the Gaussian error function (erf), which has no simple closed-form and is relatively expensive to compute on GPUs at scale; the tanh-based polynomial approximation used here matches the exact function to within about 0.1% while being much faster to evaluate.