Swish Activation Calculator
Calculate the Swish (SiLU) activation output: x times sigmoid(beta*x).
Inputs
Controls how closely Swish resembles ReLU (large β) vs. linear (β→0). Default 1.0 gives SiLU.
Swish(x)
1.761594
σ(β × x)
0.880797
Step by step
σ(β × x)
σ(1 × 2)
= 0.880797
Swish(x) = x × σ(β × x)
2 × 0.880797
= 1.761594
How it works
Swish is a smooth, self-gated activation function: Swish(x) = x·σ(β·x), where the input gates itself through a sigmoid rather than being gated by a separate learned mechanism. With β=1, Swish is identical to SiLU (Sigmoid Linear Unit). Unlike ReLU, Swish is smooth and non-monotonic (it dips slightly below zero for small negative inputs before returning to near-zero for large negative inputs), and unlike ReLU it has no hard zero cutoff, which empirically improves optimization and final accuracy in deep networks like EfficientNet, where it was introduced.
Formula
Swish(x) = x × sigma(beta × x)
- x
- Input value
- sigma
- Sigmoid function
- beta
- Shape parameter (1.0 = SiLU)
Frequently Asked Questions
What is the difference between Swish and SiLU?
SiLU is simply Swish with β fixed at exactly 1.0 — the two names are often used interchangeably in practice, since most implementations that reference 'Swish' use the β=1 variant rather than treating β as a learnable or tunable parameter.
How does beta affect the shape of Swish?
As β → 0, Swish approaches a scaled linear function (x/2); as β → ∞, Swish approaches ReLU's shape (a sharp cutoff at zero) — β=1 (SiLU) sits between these extremes as a smooth, gentle curve, and it can be made a learnable parameter to let the network adapt the activation shape during training.
Why does Swish sometimes outperform ReLU in practice?
Swish's smoothness (no sharp corner at zero) and slight non-monotonicity (allowing small negative outputs near zero) appear to improve gradient flow and optimization landscape in deep networks, which is why it was found to outperform ReLU on image classification benchmarks like ImageNet in the original EfficientNet research.