Polynomial Decay Calculator
Calculate the learning rate at a given step under a polynomial decay schedule.
Inputs
power=1 gives linear decay; power=2 gives quadratic decay.
Learning Rate at Step
5.0500 × 10⁻⁴
Step by step
Base: 1 − step ÷ total_steps
1 − 5000 ÷ 10000
= 0.500000
lr = (lr₀ − end_lr) × base^power + end_lr
(0.001 − 0.00001) × 0.5000^1 + 0.00001
= 5.0500e-4
How it works
Polynomial decay interpolates the learning rate from an initial value down to an end value using a power function of training progress: lr(step) = (lr₀ − end_lr) × (1 − step/total_steps)^power + end_lr. When power = 1, this reduces to linear decay; higher powers (e.g. 2 or 3) keep the learning rate closer to lr₀ for longer before dropping more sharply near the end of training. This schedule is notably used in the original BERT pretraining recipe (with power=1, i.e. linear decay after warmup) and remains a common choice for transformer fine-tuning.
Formula
lr(step) = (lr_0 - end_lr) × (1 - step / total_steps) ^ power + end_lr
- lr_0
- Initial learning rate
- end_lr
- End (minimum) learning rate
- step
- Current training step
- total_steps
- Total number of steps in the schedule
- power
- Polynomial exponent (1 = linear)
Frequently Asked Questions
What does the power parameter control?
It shapes the decay curve: power=1 gives a straight linear decay, power > 1 keeps the learning rate higher for longer during early-to-mid training and then decays more steeply toward the end, while power < 1 decays faster initially and flattens out near the end.
How is this different from cosine annealing?
Both interpolate smoothly between a start and end learning rate, but polynomial decay's shape is controlled by a single power exponent producing a monotonic curve, whereas cosine annealing follows the specific S-shaped cosine curve — in practice the two often produce similar-looking schedules for power values around 1–2.
What's a common use case for linear (power=1) decay?
BERT-style transformer pretraining and fine-tuning commonly use linear decay to zero (or near-zero) after a linear warmup phase, making it one of the most widely used LR schedules in NLP.
What happens if end_lr equals lr0?
The learning rate stays constant throughout training regardless of step or power, since (lr₀ − end_lr) = 0 makes the decay term vanish entirely.