Calculate the learning rate at a given step under a one-cycle policy schedule.
The one-cycle policy (Leslie Smith) runs a single cycle across the entire training run in three phases: (1) warmup, rising from a low starting LR up to a high lr_max, typically using a cosine or linear ramp; (2) decay, cosine-annealing back down from lr_max to roughly lr_min; and (3) annihilation, a final short phase dropping to an even lower lr_final for fine-grained convergence. This schedule (paired with a complementary cyclical momentum schedule in the original paper) is associated with 'super-convergence' — reaching strong accuracy in fewer epochs than constant or simple decay schedules, by tolerating a high peak learning rate for a bounded portion of training.
Warmup phase
lr = lr_min + (lr_max - lr_min) × 0.5 × (1 - cos(pi × step / warmup_steps))
Cosine decay phase
lr = lr_min + (lr_max - lr_min) × 0.5 × (1 + cos(pi × (step - warmup_steps) / decay_steps))
The high lr_max phase acts as a form of regularization and helps the optimizer escape sharp, poorly-generalizing minima early on; because it's applied briefly and framed by both a warmup and a decay, the model has time to stabilize before and after the high-LR phase rather than diverging.
Dropping to an even lower final learning rate than the initial lr_min lets the model fine-tune into a very precise minimum in the last portion of training, squeezing out additional accuracy that a stop at lr_min would leave on the table.
The original one-cycle paper recommends a learning rate range test: run a few hundred steps sweeping the LR from very low to very high and plotting loss, then pick lr_max near (but before) the point where loss starts to diverge.
The original method also cycles momentum inversely to the learning rate (low momentum during the high-LR phase, higher momentum as LR drops), though many practical implementations use one-cycle LR alone with a fixed momentum and still see strong results.