Compute the Swish (SiLU) activation function output for a given input value.
Swish is a smooth, self-gated activation function: Swish(x) = x·σ(β·x), where the input gates itself through a sigmoid rather than being gated by a separate learned mechanism. With β=1, Swish is identical to SiLU (Sigmoid Linear Unit). Unlike ReLU, Swish is smooth and non-monotonic (it dips slightly below zero for small negative inputs before returning to near-zero for large negative inputs), and unlike ReLU it has no hard zero cutoff, which empirically improves optimization and final accuracy in deep networks like EfficientNet, where it was introduced.
Swish(x) = x × sigma(beta × x)
SiLU is simply Swish with β fixed at exactly 1.0 — the two names are often used interchangeably in practice, since most implementations that reference 'Swish' use the β=1 variant rather than treating β as a learnable or tunable parameter.
As β → 0, Swish approaches a scaled linear function (x/2); as β → ∞, Swish approaches ReLU's shape (a sharp cutoff at zero) — β=1 (SiLU) sits between these extremes as a smooth, gentle curve, and it can be made a learnable parameter to let the network adapt the activation shape during training.
Swish's smoothness (no sharp corner at zero) and slight non-monotonicity (allowing small negative outputs near zero) appear to improve gradient flow and optimization landscape in deep networks, which is why it was found to outperform ReLU on image classification benchmarks like ImageNet in the original EfficientNet research.