Warmup Steps Calculator
Calculate the number of learning rate warmup steps for a training run.
Inputs
Typical range: 5%–10% of total steps.
Warmup Steps
6,000
Post-Warmup Steps
94,000
Guidance
Within the typical 5–10% range used for most transformer and CNN training runs.
Step by step
Warmup steps: total_steps × warmup_ratio
100,000 × 6.0%
= 6,000
Post-warmup steps: total_steps − warmup_steps
100,000 − 6,000
= 94,000
How it works
Learning rate warmup linearly (or otherwise) ramps the learning rate up from a small value to its peak over the first warmup_steps of training, computed as total_steps × warmup_ratio. Warmup stabilizes early training when model weights are randomly initialized and gradients can be large or noisy — without it, a high peak learning rate applied immediately can cause divergence. Typical warmup ratios are 5–10% of total training steps for transformer models; very short runs or fine-tuning jobs sometimes use even less.
Formula
warmup_steps = round(total_steps × warmup_ratio)
- total_steps
- Total number of training steps
- warmup_ratio
- Fraction of total steps spent warming up (e.g. 0.06)
Frequently Asked Questions
Why does warmup help stabilize training?
Early in training, model weights are randomly initialized and the loss surface is poorly conditioned near the initialization, so a large learning rate can cause the optimizer to take a destabilizing step; a low initial LR that gradually rises gives the model time to reach a more stable region first.
Should warmup ratio depend on batch size?
Yes, generally — larger batch sizes are often paired with higher peak learning rates (per the linear scaling rule), which increases the need for a longer warmup to avoid early instability.
Does warmup ratio depend on model size?
Larger models, especially transformers with many layers, tend to need proportionally more warmup steps because their loss landscape near initialization is more sensitive to large early updates.
What's the difference between warmup steps and the total scheduler steps used for decay?
Warmup steps are the initial ramp-up phase; the remaining (total_steps − warmup_steps) steps are typically where the primary decay schedule (cosine, linear, etc.) is applied down to a minimum learning rate.