One Cycle LR Calculator
Calculate the learning rate at a given step under a one-cycle policy schedule.
Inputs
Typically lr_max / 25.
Typically much lower than lr_min, e.g. lr_max / 100000.
Fraction of total steps spent ramping up to lr_max.
Learning Rate at Step
1.0000 × 10⁻²
Current Phase
Warmup (rising to max)
Warmup Phase Ends At Step
3,000
Decay Phase Length
6,000
Step by step
Phase 1 — Warmup: rise from lr_min to lr_max
= Steps 0–3000
Phase 2 — Decay: cosine descent from lr_max to lr_min
= Steps 3000–9000
Phase 3 — Annihilation: descent from lr_min to lr_final
= Steps 9000–10000
Current step 3000 falls in
= Warmup (rising to max): LR = 1.0000e-2
How it works
The one-cycle policy (Leslie Smith) runs a single cycle across the entire training run in three phases: (1) warmup, rising from a low starting LR up to a high lr_max, typically using a cosine or linear ramp; (2) decay, cosine-annealing back down from lr_max to roughly lr_min; and (3) annihilation, a final short phase dropping to an even lower lr_final for fine-grained convergence. This schedule (paired with a complementary cyclical momentum schedule in the original paper) is associated with 'super-convergence' — reaching strong accuracy in fewer epochs than constant or simple decay schedules, by tolerating a high peak learning rate for a bounded portion of training.
Formulas
Warmup phase
lr = lr_min + (lr_max - lr_min) × 0.5 × (1 - cos(pi × step / warmup_steps))
- lr_min
- Starting (minimum) learning rate
- lr_max
- Peak learning rate
- step
- Current step within warmup
- warmup_steps
- Total warmup steps
Cosine decay phase
lr = lr_min + (lr_max - lr_min) × 0.5 × (1 + cos(pi × (step - warmup_steps) / decay_steps))
- lr_max
- Peak learning rate
- lr_min
- Minimum learning rate
- decay_steps
- Steps in the decay phase
Frequently Asked Questions
Why does one-cycle use such a high peak learning rate?
The high lr_max phase acts as a form of regularization and helps the optimizer escape sharp, poorly-generalizing minima early on; because it's applied briefly and framed by both a warmup and a decay, the model has time to stabilize before and after the high-LR phase rather than diverging.
What's the purpose of the third 'annihilation' phase?
Dropping to an even lower final learning rate than the initial lr_min lets the model fine-tune into a very precise minimum in the last portion of training, squeezing out additional accuracy that a stop at lr_min would leave on the table.
How do I choose lr_max?
The original one-cycle paper recommends a learning rate range test: run a few hundred steps sweeping the LR from very low to very high and plotting loss, then pick lr_max near (but before) the point where loss starts to diverge.
Does one-cycle need to be paired with a momentum schedule?
The original method also cycles momentum inversely to the learning rate (low momentum during the high-LR phase, higher momentum as LR drops), though many practical implementations use one-cycle LR alone with a fixed momentum and still see strong results.