AdamW Update Calculator
Calculate the parameter update step produced by the AdamW optimizer with decoupled weight decay.
Inputs
Updated Parameter (θ_new)
0.99989428
Adam Gradient Step
0.00009572
Weight Decay Step
0.00001000
Step by step
m_t, v_t (same as Adam)
= m=0.140000, v=0.050200
m̂, v̂ (bias-corrected)
= m̂=0.214948, v̂=5.042631
Adam step: lr × m̂ ÷ (√v̂ + ε)
0.001 × 0.214948 ÷ (√5.042631 + 1e-8)
= 0.00009572
Decoupled weight decay: lr × λ × θ
0.001 × 0.01 × 1
= 0.00001000
θ_new = θ − adam_step − decay_step
1 − 0.00009572 − 0.00001000
= 0.99989428
How it works
AdamW modifies Adam by decoupling weight decay from the gradient-based adaptive update. Instead of folding weight decay into the gradient (as L2 regularization does in plain Adam, where it interacts with the adaptive per-parameter scaling in an arguably undesirable way), AdamW applies it as a separate, direct shrinkage of the parameter: θ ← θ − lr·m̂/(√v̂+ε) − lr·λ·θ, where the second term is proportional only to the learning rate, weight decay coefficient, and the parameter's own current value — not to the gradient history. This makes weight decay behave more consistently across different learning rates and is now the standard optimizer for training transformers.
Formula
theta_new = theta - lr × m_hat / (sqrt(v_hat) + epsilon) - lr × lambda × theta
- m_hat
- Bias-corrected first moment (same as Adam)
- v_hat
- Bias-corrected second moment (same as Adam)
- lr
- Learning rate
- lambda
- Decoupled weight decay coefficient
- epsilon
- Small constant for numerical stability
Frequently Asked Questions
Why is AdamW now preferred over plain Adam with L2 regularization?
In plain Adam, L2 regularization is added to the gradient before the adaptive moment estimates are computed, so parameters with large historical gradients get proportionally less weight decay — an unintended interaction. AdamW's decoupled decay applies the same proportional shrinkage regardless of gradient history, matching the behavior weight decay was originally intended to have in SGD.
Does AdamW use the same β1, β2, ε as Adam?
Yes — the moment estimation, bias correction, and adaptive scaling steps are identical to standard Adam; only the weight decay handling differs.
What's a typical weight decay value for AdamW?
0.01 is a common default for training large transformer models, though it's frequently tuned between 0.0 and 0.1 depending on model size and how much regularization is needed to prevent overfitting.
Should weight decay apply to all parameters?
Typically no — biases and normalization layer parameters (e.g. LayerNorm weight/bias) are usually excluded from weight decay in most AdamW implementations, since shrinking them doesn't serve the same regularization purpose as it does for weight matrices.