AdamW Update Calculator
Calculate the parameter update step produced by the AdamW optimizer with decoupled weight decay.
About AdamW Update Calculator
AdamW modifies Adam by decoupling weight decay from the gradient-based adaptive update. Instead of folding weight decay into the gradient (as L2 regularization does in plain Adam, where it interacts with the adaptive per-parameter scaling in an arguably undesirable way), AdamW applies it as a separate, direct shrinkage of the parameter: θ ← θ − lr·m̂/(√v̂+ε) − lr·λ·θ, where the second term is proportional only to the learning rate, weight decay coefficient, and the parameter's own current value — not to the gradient history. This makes weight decay behave more consistently across different learning rates and is now the standard optimizer for training transformers.
Formula
theta_new = theta - lr × m_hat / (sqrt(v_hat) + epsilon) - lr × lambda × theta
- m_hat
- Bias-corrected first moment (same as Adam)
- v_hat
- Bias-corrected second moment (same as Adam)
- lr
- Learning rate
- lambda
- Decoupled weight decay coefficient
- epsilon
- Small constant for numerical stability
Worked example
With the default values below, AdamW Update Calculator returns Updated Parameter (θ_new) of 0.99989428. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Current Gradient (g)
- 0.5
- Previous 1st Moment (m_{t-1})
- 0.1
- Previous 2nd Moment (v_{t-1})minimum 0
- 0.05
- Timestep (t)accepts 1–10,000,000
- 10
- Learning Rateminimum 0
- 0.001
- β1 (1st moment decay)accepts 0–1
- 0.9
- β2 (2nd moment decay)accepts 0–1
- 0.999
- ε (stability constant)minimum 0
- 0
- Current Parameter (θ)
- 1
- Weight Decay (λ)minimum 0
- 0.01
Results
- Updated Parameter (θ_new)
- 0.99989428
- Adam Gradient Step
- 0.00009572
- Weight Decay Step
- 0.00001000
Step by step
- 1m_t, v_t (same as Adam)= m=0.140000, v=0.050200
- 2m̂, v̂ (bias-corrected)= m̂=0.214948, v̂=5.042631
- 3Adam step: lr × m̂ ÷ (√v̂ + ε)= 0.00009572
- 4Decoupled weight decay: lr × λ × θ= 0.00001000
- 5θ_new = θ − adam_step − decay_step= 0.99989428
Assumptions and limits
- The result is derived only from the 10 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 8 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
Why is AdamW now preferred over plain Adam with L2 regularization?
In plain Adam, L2 regularization is added to the gradient before the adaptive moment estimates are computed, so parameters with large historical gradients get proportionally less weight decay — an unintended interaction. AdamW's decoupled decay applies the same proportional shrinkage regardless of gradient history, matching the behavior weight decay was originally intended to have in SGD.
Does AdamW use the same β1, β2, ε as Adam?
Yes — the moment estimation, bias correction, and adaptive scaling steps are identical to standard Adam; only the weight decay handling differs.
What's a typical weight decay value for AdamW?
0.01 is a common default for training large transformer models, though it's frequently tuned between 0.0 and 0.1 depending on model size and how much regularization is needed to prevent overfitting.
Should weight decay apply to all parameters?
Typically no — biases and normalization layer parameters (e.g. LayerNorm weight/bias) are usually excluded from weight decay in most AdamW implementations, since shrinking them doesn't serve the same regularization purpose as it does for weight matrices.