Calculate the parameter update step produced by the AdamW optimizer with decoupled weight decay.
AdamW modifies Adam by decoupling weight decay from the gradient-based adaptive update. Instead of folding weight decay into the gradient (as L2 regularization does in plain Adam, where it interacts with the adaptive per-parameter scaling in an arguably undesirable way), AdamW applies it as a separate, direct shrinkage of the parameter: θ ← θ − lr·m̂/(√v̂+ε) − lr·λ·θ, where the second term is proportional only to the learning rate, weight decay coefficient, and the parameter's own current value — not to the gradient history. This makes weight decay behave more consistently across different learning rates and is now the standard optimizer for training transformers.
theta_new = theta - lr × m_hat / (sqrt(v_hat) + epsilon) - lr × lambda × theta
In plain Adam, L2 regularization is added to the gradient before the adaptive moment estimates are computed, so parameters with large historical gradients get proportionally less weight decay — an unintended interaction. AdamW's decoupled decay applies the same proportional shrinkage regardless of gradient history, matching the behavior weight decay was originally intended to have in SGD.
Yes — the moment estimation, bias correction, and adaptive scaling steps are identical to standard Adam; only the weight decay handling differs.
0.01 is a common default for training large transformer models, though it's frequently tuned between 0.0 and 0.1 depending on model size and how much regularization is needed to prevent overfitting.
Typically no — biases and normalization layer parameters (e.g. LayerNorm weight/bias) are usually excluded from weight decay in most AdamW implementations, since shrinking them doesn't serve the same regularization purpose as it does for weight matrices.