Calculate the learning rate at a given training step for common LR schedules.
Learning rate schedules control how the learning rate changes during training to balance fast early progress with stable convergence later. Step decay drops the rate by a fixed factor at regular intervals. Exponential decay shrinks it continuously by a fixed multiplicative rate each epoch. Cosine annealing smoothly decreases the rate following a cosine curve from the initial value down to near zero. Warmup + cosine linearly increases the rate for a few initial epochs (helping stabilize early training, especially for large batch sizes or transformers) before switching to cosine decay for the remainder of training.
LR_cosine = initial_LR * 0.5 * (1 + cos(pi * epoch / total_epochs))
Warmup prevents large, unstable weight updates early in training when gradients and optimizer statistics (like Adam's moment estimates) haven't yet stabilized, which is especially important for large batch sizes and transformer architectures.
Warmup followed by cosine decay is the de facto standard for training large transformer models, as popularized by GPT and BERT-style training recipes.
Step decay drops the learning rate abruptly at fixed epoch intervals, while exponential decay reduces it smoothly and continuously every epoch — step decay is simpler to reason about, but exponential/cosine schedules often give smoother convergence.
It approaches but reaches exactly zero only at the very last epoch of the schedule, when the cosine term equals -1, making LR = initial_LR × 0.5 × (1 + (-1)) = 0.