Polynomial Decay Calculator
Calculate the learning rate at a given step under a polynomial decay schedule.
About Polynomial Decay Calculator
Polynomial decay interpolates the learning rate from an initial value down to an end value using a power function of training progress: lr(step) = (lr₀ − end_lr) × (1 − step/total_steps)^power + end_lr. When power = 1, this reduces to linear decay; higher powers (e.g. 2 or 3) keep the learning rate closer to lr₀ for longer before dropping more sharply near the end of training. This schedule is notably used in the original BERT pretraining recipe (with power=1, i.e. linear decay after warmup) and remains a common choice for transformer fine-tuning.
Formula
lr(step) = (lr_0 - end_lr) × (1 - step / total_steps) ^ power + end_lr
- lr_0
- Initial learning rate
- end_lr
- End (minimum) learning rate
- step
- Current training step
- total_steps
- Total number of steps in the schedule
- power
- Polynomial exponent (1 = linear)
Worked example
With the default values below, Polynomial Decay Calculator returns Learning Rate at Step of 5.0500 × 10⁻⁴. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Initial Learning Rateminimum 0
- 0.001
- End Learning Rateminimum 0
- 0.00001
- Current Stepaccepts 0–1,000,000,000
- 5,000
- Total Stepsaccepts 1–1,000,000,000
- 10,000
- Poweraccepts 0.1–10
- 1
Results
- Learning Rate at Step
- 5.0500 × 10⁻⁴
Step by step
- 1Base: 1 − step ÷ total_steps= 0.500000
- 2lr = (lr₀ − end_lr) × base^power + end_lr= 5.0500e-4
Assumptions and limits
- The result is derived only from the 5 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 4 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
What does the power parameter control?
It shapes the decay curve: power=1 gives a straight linear decay, power > 1 keeps the learning rate higher for longer during early-to-mid training and then decays more steeply toward the end, while power < 1 decays faster initially and flattens out near the end.
How is this different from cosine annealing?
Both interpolate smoothly between a start and end learning rate, but polynomial decay's shape is controlled by a single power exponent producing a monotonic curve, whereas cosine annealing follows the specific S-shaped cosine curve — in practice the two often produce similar-looking schedules for power values around 1–2.
What's a common use case for linear (power=1) decay?
BERT-style transformer pretraining and fine-tuning commonly use linear decay to zero (or near-zero) after a linear warmup phase, making it one of the most widely used LR schedules in NLP.
What happens if end_lr equals lr0?
The learning rate stays constant throughout training regardless of step or power, since (lr₀ − end_lr) = 0 makes the decay term vanish entirely.