Compute the Softplus activation function, a smooth approximation of ReLU.
Softplus is a smooth approximation of ReLU: Softplus(x) = ln(1 + e^x), which behaves almost identically to max(0, x) for inputs far from zero, but replaces ReLU's sharp corner with a smooth, continuously differentiable curve. Its derivative is exactly the sigmoid function, softplus'(x) = σ(x), which is a convenient closed form. Because softplus is always positive and never exactly zero, it's commonly used to parameterize quantities that must stay strictly positive (like a standard deviation in a probabilistic model), and it also appears as a building block inside the Mish activation function.
Softplus(x) = ln(1 + e^x)
Softplus is a smooth, differentiable approximation of ReLU — for large positive or negative x, Softplus(x) closely matches max(0,x), but near x=0 it transitions smoothly instead of having ReLU's sharp corner, and it never outputs exactly 0 (it asymptotically approaches 0 for very negative x).
This falls directly out of calculus: differentiating ln(1+e^x) gives e^x/(1+e^x), which is algebraically identical to 1/(1+e^(−x)) — the sigmoid function — making softplus and sigmoid a mathematically paired activation/derivative combination.
It's commonly used to constrain a neural network's output to be strictly positive (e.g. predicting a variance or scale parameter in a probabilistic model, since variance can't be negative), and it's also a core building block inside the Mish activation function.