Learning Rate Scheduler Calculator
Calculate the learning rate at a given training step for common LR schedules.
About Learning Rate Scheduler Calculator
Learning rate schedules control how the learning rate changes during training to balance fast early progress with stable convergence later. Step decay drops the rate by a fixed factor at regular intervals. Exponential decay shrinks it continuously by a fixed multiplicative rate each epoch. Cosine annealing smoothly decreases the rate following a cosine curve from the initial value down to near zero. Warmup + cosine linearly increases the rate for a few initial epochs (helping stabilize early training, especially for large batch sizes or transformers) before switching to cosine decay for the remainder of training.
Formula
LR_cosine = initial_LR * 0.5 * (1 + cos(pi * epoch / total_epochs))
- \eta_0
- Initial (peak) learning rate
- t
- Current epoch
- T
- Total epochs
Worked example
With the default values below, Learning Rate Scheduler Calculator returns Learning Rate at Current Epoch of 5.0000 × 10⁻⁴. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Initial Learning Rateaccepts 0–10
- 0.001
- Schedule Typeone of Step Decay, Exponential Decay, Cosine Annealing, Warmup + Cosine
- Cosine Annealing
- Total Epochsaccepts 1–10,000 epochs
- 50 epochs
- Current Epochaccepts 0–10,000 epoch
- 25 epoch
- Warmup Epochsaccepts 0–1,000 epochs
- 5 epochs
Results
- Learning Rate at Current Epoch
- 5.0000 × 10⁻⁴
- Learning Rate at Final Epoch
- 0
Step by step
- 1Schedule formula (cosine)= LR(25) = 5.0000e-4
- 2Learning rate at final epoch= 0.0000e+0
Assumptions and limits
- The result is derived only from the 5 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 4 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
Why use a warmup period at all?
Warmup prevents large, unstable weight updates early in training when gradients and optimizer statistics (like Adam's moment estimates) haven't yet stabilized, which is especially important for large batch sizes and transformer architectures.
Which schedule is most commonly used for LLM training?
Warmup followed by cosine decay is the de facto standard for training large transformer models, as popularized by GPT and BERT-style training recipes.
What's the difference between step decay and exponential decay?
Step decay drops the learning rate abruptly at fixed epoch intervals, while exponential decay reduces it smoothly and continuously every epoch — step decay is simpler to reason about, but exponential/cosine schedules often give smoother convergence.
Does the learning rate ever reach exactly zero with cosine annealing?
It approaches but reaches exactly zero only at the very last epoch of the schedule, when the cosine term equals -1, making LR = initial_LR × 0.5 × (1 + (-1)) = 0.