Adam Optimizer Update Calculator
Calculate the parameter update step produced by the Adam optimizer.
Inputs
Updated Parameter (θ_new)
0.99990428
Update Magnitude
0.00009572
Bias-Corrected 1st Moment (m̂)
0.214948
Bias-Corrected 2nd Moment (v̂)
5.042631
Raw 1st Moment (m_t)
0.140000
Raw 2nd Moment (v_t)
0.050200
Step by step
m_t = β1×m_{t-1} + (1−β1)×g
0.9×0.1 + 0.100×0.5
= 0.140000
v_t = β2×v_{t-1} + (1−β2)×g²
0.999×0.05 + 0.0010×0.5²
= 0.050200
m̂ = m_t ÷ (1 − β1^t)
0.140000 ÷ (1 − 0.9^10)
= 0.214948
v̂ = v_t ÷ (1 − β2^t)
0.050200 ÷ (1 − 0.999^10)
= 5.042631
Update = lr × m̂ ÷ (√v̂ + ε)
0.001 × 0.214948 ÷ (√5.042631 + 1e-8)
= 0.00009572
θ_new = θ − update
1 − 0.00009572
= 0.99990428
How it works
Adam (Adaptive Moment Estimation) maintains exponentially decaying averages of past gradients (first moment m, like momentum) and past squared gradients (second moment v, like RMSProp), then bias-corrects both before computing the update: m_t = β1·m_{t-1} + (1−β1)·g, v_t = β2·v_{t-1} + (1−β2)·g², with bias-corrected estimates m̂ = m_t/(1−β1^t) and v̂ = v_t/(1−β2^t) that counteract the moments' initialization at zero (which biases early estimates toward zero, especially at low t). The final update is θ ← θ − lr·m̂/(√v̂ + ε), where ε prevents division by zero. Adam combines the benefits of momentum (smoothing gradient direction) and adaptive per-parameter learning rates (scaling by gradient magnitude history).
Formula
Adam update rule
theta_new = theta - lr × m_hat / (sqrt(v_hat) + epsilon)
- m_hat
- Bias-corrected first moment: m_t / (1 - beta1^t)
- v_hat
- Bias-corrected second moment: v_t / (1 - beta2^t)
- lr
- Learning rate
- epsilon
- Small constant for numerical stability
Frequently Asked Questions
Why does bias correction matter more at small t?
Since m and v are both initialized to zero, early estimates are heavily biased toward zero before enough gradient history accumulates; the bias-correction terms (1−β1^t) and (1−β2^t) are small at low t, which inflates m̂ and v̂ to counteract this — as t grows, β1^t and β2^t approach zero and the correction becomes negligible.
What do typical β1 and β2 values represent?
β1 = 0.9 means the first moment averages over roughly the last 10 gradients (a short memory, similar to standard momentum); β2 = 0.999 means the second moment averages over roughly the last 1000 gradients (a much longer memory), which stabilizes the adaptive learning rate scaling.
What does epsilon (ε) do?
ε is a small constant (typically 1e-8) added to the denominator purely to prevent division by zero when v̂ is very close to zero, and has negligible effect on the update otherwise.
How does Adam differ from AdamW?
Plain Adam applies weight decay as an L2 penalty baked into the gradient itself, which interacts with the adaptive learning rate in an arguably undesirable way; AdamW instead subtracts weight decay directly from the parameters as a separate step, decoupled from the adaptive gradient scaling — see the AdamW Update Calculator.