Adam Optimizer Update Calculator
Calculate the parameter update step produced by the Adam optimizer.
About Adam Optimizer Update Calculator
Adam (Adaptive Moment Estimation) maintains exponentially decaying averages of past gradients (first moment m, like momentum) and past squared gradients (second moment v, like RMSProp), then bias-corrects both before computing the update: m_t = β1·m_{t-1} + (1−β1)·g, v_t = β2·v_{t-1} + (1−β2)·g², with bias-corrected estimates m̂ = m_t/(1−β1^t) and v̂ = v_t/(1−β2^t) that counteract the moments' initialization at zero (which biases early estimates toward zero, especially at low t). The final update is θ ← θ − lr·m̂/(√v̂ + ε), where ε prevents division by zero. Adam combines the benefits of momentum (smoothing gradient direction) and adaptive per-parameter learning rates (scaling by gradient magnitude history).
Formula
Adam update rule
theta_new = theta - lr × m_hat / (sqrt(v_hat) + epsilon)
- m_hat
- Bias-corrected first moment: m_t / (1 - beta1^t)
- v_hat
- Bias-corrected second moment: v_t / (1 - beta2^t)
- lr
- Learning rate
- epsilon
- Small constant for numerical stability
Worked example
With the default values below, Adam Optimizer Update Calculator returns Updated Parameter (θ_new) of 0.99990428. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Current Gradient (g)
- 0.5
- Previous 1st Moment (m_{t-1})
- 0.1
- Previous 2nd Moment (v_{t-1})minimum 0
- 0.05
- Timestep (t)accepts 1–10,000,000
- 10
- Learning Rateminimum 0
- 0.001
- β1 (1st moment decay)accepts 0–1
- 0.9
- β2 (2nd moment decay)accepts 0–1
- 0.999
- ε (stability constant)minimum 0
- 0
- Current Parameter (θ)
- 1
Results
- Updated Parameter (θ_new)
- 0.99990428
- Update Magnitude
- 0.00009572
- Bias-Corrected 1st Moment (m̂)
- 0.214948
- Bias-Corrected 2nd Moment (v̂)
- 5.042631
- Raw 1st Moment (m_t)
- 0.140000
- Raw 2nd Moment (v_t)
- 0.050200
Step by step
- 1m_t = β1×m_{t-1} + (1−β1)×g= 0.140000
- 2v_t = β2×v_{t-1} + (1−β2)×g²= 0.050200
- 3m̂ = m_t ÷ (1 − β1^t)= 0.214948
- 4v̂ = v_t ÷ (1 − β2^t)= 5.042631
- 5Update = lr × m̂ ÷ (√v̂ + ε)= 0.00009572
- 6θ_new = θ − update= 0.99990428
Assumptions and limits
- The result is derived only from the 9 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 8 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
Why does bias correction matter more at small t?
Since m and v are both initialized to zero, early estimates are heavily biased toward zero before enough gradient history accumulates; the bias-correction terms (1−β1^t) and (1−β2^t) are small at low t, which inflates m̂ and v̂ to counteract this — as t grows, β1^t and β2^t approach zero and the correction becomes negligible.
What do typical β1 and β2 values represent?
β1 = 0.9 means the first moment averages over roughly the last 10 gradients (a short memory, similar to standard momentum); β2 = 0.999 means the second moment averages over roughly the last 1000 gradients (a much longer memory), which stabilizes the adaptive learning rate scaling.
What does epsilon (ε) do?
ε is a small constant (typically 1e-8) added to the denominator purely to prevent division by zero when v̂ is very close to zero, and has negligible effect on the update otherwise.
How does Adam differ from AdamW?
Plain Adam applies weight decay as an L2 penalty baked into the gradient itself, which interacts with the adaptive learning rate in an arguably undesirable way; AdamW instead subtracts weight decay directly from the parameters as a separate step, decoupled from the adaptive gradient scaling — see the AdamW Update Calculator.