Compute the hyperbolic tangent activation function output for a given input.
Tanh squashes real-valued inputs into the range (−1, 1): tanh(x) = (e^x − e^(−x)) / (e^x + e^(−x)), and is simply a rescaled and shifted version of the sigmoid function (tanh(x) = 2·σ(2x) − 1). Its key advantage over sigmoid is being zero-centered, meaning outputs can be negative, which helps keep gradients balanced during training. Like sigmoid, its derivative, 1 − tanh(x)², is at most 1 (at x=0) and shrinks toward 0 for large |x|, so tanh still suffers from vanishing gradients in very deep networks, but it remains a standard choice for RNN/LSTM hidden state activations.
Tanh function
tanh(x) = (e^x - e^(-x)) / (e^x + e^(-x))
Tanh derivative
tanh'(x) = 1 - tanh(x)^2
Tanh is a rescaled sigmoid: tanh(x) = 2σ(2x) − 1, which maps sigmoid's (0,1) output range to (−1,1) — both share the same S-shaped curve, but tanh's zero-centering often makes it converge faster in practice for hidden layers.
When an activation function's outputs are all positive (like sigmoid's), gradients for the next layer's weights tend to all move in the same direction, slowing convergence; tanh's symmetric (−1,1) output allows both positive and negative signals, generally leading to more balanced gradient updates.
Tanh remains standard for the hidden state and gate activations inside LSTM and GRU recurrent cells, and appears in the GELU approximation formula, though it has largely been replaced by ReLU-family functions in standard feedforward and convolutional hidden layers.