Skip to content
ENExplore Calculators

ReLU Activation Output Calculator

Work out relu activation output instantly with clear inputs, formula shown and shareable results.

About ReLU Activation Output Calculator

ReLU passes positive values unchanged and clamps negatives to zero, which keeps gradients at exactly 1 in the active region and avoids the vanishing-gradient problem of sigmoid and tanh. Its weakness is that a unit stuck at negative pre-activation has zero gradient forever. Leaky ReLU keeps a small slope alpha there, and ELU uses a smooth exponential branch that also centres activations nearer zero.

Formula

Activation definitions

ReLU: f(x) = max(0, x); Leaky: f(x) = x if x > 0 else alpha x; ELU: f(x) = x if x > 0 else alpha(e^x - 1)

Worked example

With the default values below, ReLU Activation Output Calculator returns Activation output of -0.01500. Change any input above and every figure here is recalculated in your browser.

Inputs used

Pre-activation value
-1.5
Negative slope / alphaaccepts 0–1
0.01
Activation variantone of ReLU: max(0, x), Leaky ReLU: max(alpha x, x), ELU: alpha(e^x - 1) for x < 0
Leaky ReLU: max(alpha x, x)

Results

Activation output
-0.01500
Local gradient
0.01000
Dead-unit risk
No - gradient is non-zero
Variant applied
Leaky ReLU (alpha = 0.01)

Step by step

  1. 1Values used
  2. 2Activation definitions
  3. 3Activation output= -0.01500
  4. 4Local gradient= 0.01000
  5. 5Dead-unit risk= No - gradient is non-zero
  6. 6Variant applied= Leaky ReLU (alpha = 0.01)

Assumptions and limits

  • The result is derived only from the 3 inputs listed above. Anything not among them is not part of the calculation.
  • In particular, this calculator does not take inflation, fees or charges or compounding frequency into account.
  • Figures are rounded for display to 5 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.

How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.

Frequently Asked Questions

What causes dying ReLU?

A large negative bias or an aggressive learning rate can push a unit permanently into the negative region. Since the gradient there is exactly zero, no update can revive it.

Which variant should I default to?

Plain ReLU is still a strong default for convolutional networks. Transformers mostly use GELU or SwiGLU, which are smooth and slightly better empirically at the same cost.

Related Calculators

ReLU Activation Output Calculator | Calcrivo