Calculate the parameter update step produced by the Adam optimizer.
Adam (Adaptive Moment Estimation) maintains exponentially decaying averages of past gradients (first moment m, like momentum) and past squared gradients (second moment v, like RMSProp), then bias-corrects both before computing the update: m_t = β1·m_{t-1} + (1−β1)·g, v_t = β2·v_{t-1} + (1−β2)·g², with bias-corrected estimates m̂ = m_t/(1−β1^t) and v̂ = v_t/(1−β2^t) that counteract the moments' initialization at zero (which biases early estimates toward zero, especially at low t). The final update is θ ← θ − lr·m̂/(√v̂ + ε), where ε prevents division by zero. Adam combines the benefits of momentum (smoothing gradient direction) and adaptive per-parameter learning rates (scaling by gradient magnitude history).
Adam update rule
theta_new = theta - lr × m_hat / (sqrt(v_hat) + epsilon)
Since m and v are both initialized to zero, early estimates are heavily biased toward zero before enough gradient history accumulates; the bias-correction terms (1−β1^t) and (1−β2^t) are small at low t, which inflates m̂ and v̂ to counteract this — as t grows, β1^t and β2^t approach zero and the correction becomes negligible.
β1 = 0.9 means the first moment averages over roughly the last 10 gradients (a short memory, similar to standard momentum); β2 = 0.999 means the second moment averages over roughly the last 1000 gradients (a much longer memory), which stabilizes the adaptive learning rate scaling.
ε is a small constant (typically 1e-8) added to the denominator purely to prevent division by zero when v̂ is very close to zero, and has negligible effect on the update otherwise.
Plain Adam applies weight decay as an L2 penalty baked into the gradient itself, which interacts with the adaptive learning rate in an arguably undesirable way; AdamW instead subtracts weight decay directly from the parameters as a separate step, decoupled from the adaptive gradient scaling — see the AdamW Update Calculator.