Calculate the number of learning rate warmup steps for a training run.
Learning rate warmup linearly (or otherwise) ramps the learning rate up from a small value to its peak over the first warmup_steps of training, computed as total_steps × warmup_ratio. Warmup stabilizes early training when model weights are randomly initialized and gradients can be large or noisy — without it, a high peak learning rate applied immediately can cause divergence. Typical warmup ratios are 5–10% of total training steps for transformer models; very short runs or fine-tuning jobs sometimes use even less.
warmup_steps = round(total_steps × warmup_ratio)
Early in training, model weights are randomly initialized and the loss surface is poorly conditioned near the initialization, so a large learning rate can cause the optimizer to take a destabilizing step; a low initial LR that gradually rises gives the model time to reach a more stable region first.
Yes, generally — larger batch sizes are often paired with higher peak learning rates (per the linear scaling rule), which increases the need for a longer warmup to avoid early instability.
Larger models, especially transformers with many layers, tend to need proportionally more warmup steps because their loss landscape near initialization is more sensitive to large early updates.
Warmup steps are the initial ramp-up phase; the remaining (total_steps − warmup_steps) steps are typically where the primary decay schedule (cosine, linear, etc.) is applied down to a minimum learning rate.