Calculate the learning rate at a given step under a polynomial decay schedule.
Polynomial decay interpolates the learning rate from an initial value down to an end value using a power function of training progress: lr(step) = (lr₀ − end_lr) × (1 − step/total_steps)^power + end_lr. When power = 1, this reduces to linear decay; higher powers (e.g. 2 or 3) keep the learning rate closer to lr₀ for longer before dropping more sharply near the end of training. This schedule is notably used in the original BERT pretraining recipe (with power=1, i.e. linear decay after warmup) and remains a common choice for transformer fine-tuning.
lr(step) = (lr_0 - end_lr) × (1 - step / total_steps) ^ power + end_lr
It shapes the decay curve: power=1 gives a straight linear decay, power > 1 keeps the learning rate higher for longer during early-to-mid training and then decays more steeply toward the end, while power < 1 decays faster initially and flattens out near the end.
Both interpolate smoothly between a start and end learning rate, but polynomial decay's shape is controlled by a single power exponent producing a monotonic curve, whereas cosine annealing follows the specific S-shaped cosine curve — in practice the two often produce similar-looking schedules for power values around 1–2.
BERT-style transformer pretraining and fine-tuning commonly use linear decay to zero (or near-zero) after a linear warmup phase, making it one of the most widely used LR schedules in NLP.
The learning rate stays constant throughout training regardless of step or power, since (lr₀ − end_lr) = 0 makes the decay term vanish entirely.