Weight Decay Calculator
Calculate the effective weight decay applied during regularized training.
About Weight Decay Calculator
Weight decay shrinks model weights toward zero at every optimizer step, acting as L2 regularization to reduce overfitting. Classic L2 regularization adds a penalty term wd × Σw² / 2 to the loss, whose gradient effectively scales weights by (1 − lr×wd) each step — coupling weight decay to the learning rate, shown here as effective_lr = lr × (1 − wd). AdamW decouples weight decay from the adaptive learning rate entirely (see the AdamW Update Calculator), which is why AdamW's decay behaves more predictably across different learning rates than plain L2 regularization in Adam.
Formulas
L2 penalty term
L2_penalty = (wd × sum(w^2)) / 2
- wd
- Weight decay coefficient (lambda)
- w
- Model weight parameters
Per-step weight retention
decay_factor = 1 - lr × wd
- lr
- Learning rate
- wd
- Weight decay coefficient
Worked example
With the default values below, Weight Decay Calculator returns Effective Learning Rate of 0.00099000. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Learning Rateaccepts 0–10
- 0.001
- Weight Decay (λ)accepts 0–10
- 0.01
- Sum of Squared Weights (Σw²)accepts 0–1,000,000,000,000
- 500
Results
- Effective Learning Rate
- 0.00099000
- L2 Penalty Term
- 2.500000
- Per-Step Weight Retention Factor
- 0.99999000
Step by step
- 1Effective LR (SGD-style coupled decay): lr × (1 − wd)= 9.9000e-4
- 2L2 penalty: wd × Σw² ÷ 2= 2.500000
- 3Per-step weight retention factor: 1 − lr×wd= 0.99999000
Assumptions and limits
- The result is derived only from the 3 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 8 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
What's a typical weight decay value?
Common values range from 0.0 (no decay) to 0.1, with 0.01 being a frequent default for AdamW on transformer models; higher values regularize more aggressively but can underfit if set too high.
Why does 'effective learning rate' matter here?
In optimizers where weight decay is coupled to the learning rate (like L2 regularization inside vanilla Adam), a change in the learning rate schedule also changes the effective regularization strength over time — this is one of the core motivations behind AdamW's decoupled decay.
Should weight decay be applied to bias terms?
Typically no — most implementations exclude biases and normalization layer parameters (e.g. LayerNorm scale/shift) from weight decay, since shrinking them toward zero doesn't serve the same regularization purpose as it does for weight matrices.
How does weight decay interact with dropout?
They are complementary regularizers — dropout reduces overfitting by randomly zeroing activations during training, while weight decay directly penalizes large weight magnitudes; many architectures use both simultaneously.