Cosine Annealing Calculator
Calculate the learning rate at a given step under a cosine annealing schedule.
Inputs
Learning Rate at Step t
5.0500 × 10⁻⁴
Schedule Progress
50.0%
Step by step
cos(π × t ÷ T)
cos(π × 5000 ÷ 10000)
= 0.000000
lr = lr_min + 0.5×(lr_max−lr_min)×(1+cos)
0.00001 + 0.5×(0.001−0.00001)×(1+0.0000)
= 5.0500e-4
How it works
Cosine annealing smoothly decays the learning rate following a cosine curve from lr_max down to lr_min over T steps: lr(t) = lr_min + 0.5 × (lr_max − lr_min) × (1 + cos(π·t/T)). At t=0, cos(0)=1, giving lr_max; at t=T, cos(π)=−1, giving lr_min. The curve decreases slowly at first, accelerates through the middle of training, and flattens out again near the end — this shape empirically tends to help models settle into a good minimum more smoothly than a linear or step decay, and is a core component of the popular SGDR (Stochastic Gradient Descent with Warm Restarts) and one-cycle training schedules.
Formula
lr(t) = lr_min + 0.5 × (lr_max - lr_min) × (1 + cos(pi × t / T))
- lr_min
- Minimum learning rate
- lr_max
- Maximum learning rate
- t
- Current step
- T
- Total steps in the schedule
Frequently Asked Questions
Why is cosine annealing popular for training deep models?
Its smooth, non-linear decay spends more time at higher learning rates early (encouraging fast initial progress) and tapers gradually near the end (allowing fine-grained convergence), which empirically often outperforms step decay or pure exponential decay for many vision and NLP training regimes.
What happens after t exceeds T?
The plain cosine annealing formula is only defined for t in [0, T]; this calculator clamps t to T, effectively holding the learning rate at lr_min once the schedule's horizon has passed — some implementations instead restart the cosine cycle (SGDR) or switch to a different post-schedule strategy.
How does this combine with warmup?
It's common to apply a linear warmup for the first 5–10% of steps before switching to cosine annealing for the remainder — the warmup steps are typically excluded from T, i.e. T represents only the post-warmup decay phase.
How is this different from one-cycle LR?
One-cycle LR combines an explicit warmup phase (ramping up to lr_max) with a cosine (or linear) decay phase, all in a single schedule; this calculator computes only the decay portion, assuming the ramp-up has already happened or isn't used.