Calculate the effective weight decay applied during regularized training.
Weight decay shrinks model weights toward zero at every optimizer step, acting as L2 regularization to reduce overfitting. Classic L2 regularization adds a penalty term wd × Σw² / 2 to the loss, whose gradient effectively scales weights by (1 − lr×wd) each step — coupling weight decay to the learning rate, shown here as effective_lr = lr × (1 − wd). AdamW decouples weight decay from the adaptive learning rate entirely (see the AdamW Update Calculator), which is why AdamW's decay behaves more predictably across different learning rates than plain L2 regularization in Adam.
L2 penalty term
L2_penalty = (wd × sum(w^2)) / 2
Per-step weight retention
decay_factor = 1 - lr × wd
Common values range from 0.0 (no decay) to 0.1, with 0.01 being a frequent default for AdamW on transformer models; higher values regularize more aggressively but can underfit if set too high.
In optimizers where weight decay is coupled to the learning rate (like L2 regularization inside vanilla Adam), a change in the learning rate schedule also changes the effective regularization strength over time — this is one of the core motivations behind AdamW's decoupled decay.
Typically no — most implementations exclude biases and normalization layer parameters (e.g. LayerNorm scale/shift) from weight decay, since shrinking them toward zero doesn't serve the same regularization purpose as it does for weight matrices.
They are complementary regularizers — dropout reduces overfitting by randomly zeroing activations during training, while weight decay directly penalizes large weight magnitudes; many architectures use both simultaneously.