Learning Rate Scheduler Calculator
Visualize how the learning rate changes over training epochs for common LR schedules.
Inputs
Peak learning rate used at the start (or after warmup).
The epoch at which to evaluate the learning rate.
Only used for Warmup + Cosine schedule.
Learning Rate at Current Epoch
5.0000 × 10⁻⁴
Learning Rate at Final Epoch
0
Step by step
Schedule formula (cosine)
LR = initial_LR × 0.5 × (1 + cos(π × epoch / total_epochs))
= LR(25) = 5.0000e-4
Learning rate at final epoch
LR(50)
= 0.0000e+0
How it works
Learning rate schedules control how the learning rate changes during training to balance fast early progress with stable convergence later. Step decay drops the rate by a fixed factor at regular intervals. Exponential decay shrinks it continuously by a fixed multiplicative rate each epoch. Cosine annealing smoothly decreases the rate following a cosine curve from the initial value down to near zero. Warmup + cosine linearly increases the rate for a few initial epochs (helping stabilize early training, especially for large batch sizes or transformers) before switching to cosine decay for the remainder of training.
Formula
LR_cosine = initial_LR * 0.5 * (1 + cos(pi * epoch / total_epochs))
- \eta_0
- Initial (peak) learning rate
- t
- Current epoch
- T
- Total epochs
Frequently Asked Questions
Why use a warmup period at all?
Warmup prevents large, unstable weight updates early in training when gradients and optimizer statistics (like Adam's moment estimates) haven't yet stabilized, which is especially important for large batch sizes and transformer architectures.
Which schedule is most commonly used for LLM training?
Warmup followed by cosine decay is the de facto standard for training large transformer models, as popularized by GPT and BERT-style training recipes.
What's the difference between step decay and exponential decay?
Step decay drops the learning rate abruptly at fixed epoch intervals, while exponential decay reduces it smoothly and continuously every epoch — step decay is simpler to reason about, but exponential/cosine schedules often give smoother convergence.
Does the learning rate ever reach exactly zero with cosine annealing?
It approaches but reaches exactly zero only at the very last epoch of the schedule, when the cosine term equals -1, making LR = initial_LR × 0.5 × (1 + (-1)) = 0.