Weight Decay Calculator
Calculate the effective weight decay applied during regularized training.
Inputs
L2 regularization coefficient, e.g. 0.01 for AdamW.
Used only to illustrate the L2 penalty term; optional for the LR-scaling result.
Effective Learning Rate
0.00099000
L2 Penalty Term
2.500000
Per-Step Weight Retention Factor
0.99999000
Step by step
Effective LR (SGD-style coupled decay): lr × (1 − wd)
0.001 × (1 − 0.01)
= 9.9000e-4
L2 penalty: wd × Σw² ÷ 2
0.01 × 500 ÷ 2
= 2.500000
Per-step weight retention factor: 1 − lr×wd
1 − 0.001×0.01
= 0.99999000
How it works
Weight decay shrinks model weights toward zero at every optimizer step, acting as L2 regularization to reduce overfitting. Classic L2 regularization adds a penalty term wd × Σw² / 2 to the loss, whose gradient effectively scales weights by (1 − lr×wd) each step — coupling weight decay to the learning rate, shown here as effective_lr = lr × (1 − wd). AdamW decouples weight decay from the adaptive learning rate entirely (see the AdamW Update Calculator), which is why AdamW's decay behaves more predictably across different learning rates than plain L2 regularization in Adam.
Formulas
L2 penalty term
L2_penalty = (wd × sum(w^2)) / 2
- wd
- Weight decay coefficient (lambda)
- w
- Model weight parameters
Per-step weight retention
decay_factor = 1 - lr × wd
- lr
- Learning rate
- wd
- Weight decay coefficient
Frequently Asked Questions
What's a typical weight decay value?
Common values range from 0.0 (no decay) to 0.1, with 0.01 being a frequent default for AdamW on transformer models; higher values regularize more aggressively but can underfit if set too high.
Why does 'effective learning rate' matter here?
In optimizers where weight decay is coupled to the learning rate (like L2 regularization inside vanilla Adam), a change in the learning rate schedule also changes the effective regularization strength over time — this is one of the core motivations behind AdamW's decoupled decay.
Should weight decay be applied to bias terms?
Typically no — most implementations exclude biases and normalization layer parameters (e.g. LayerNorm scale/shift) from weight decay, since shrinking them toward zero doesn't serve the same regularization purpose as it does for weight matrices.
How does weight decay interact with dropout?
They are complementary regularizers — dropout reduces overfitting by randomly zeroing activations during training, while weight decay directly penalizes large weight magnitudes; many architectures use both simultaneously.