Cyclic Learning Rate Calculator
Calculate the learning rate at a given step under a cyclic learning rate schedule.
About Cyclic Learning Rate Calculator
Cyclic learning rate (CLR) policies oscillate the learning rate between a base_lr and max_lr repeatedly throughout training instead of monotonically decaying. The 'triangular' policy is a simple linear ramp up and down every 2×step_size steps. 'Triangular2' halves the amplitude (max_lr − base_lr) after every full cycle, giving progressively smaller oscillations. 'Exp_range' instead shrinks the amplitude exponentially with each step via a decay constant gamma (close to but below 1). Cycling the learning rate lets training periodically revisit higher LRs, which can help escape sharp local minima and reduces the need to manually tune a single optimal learning rate.
Formula
lr = base_lr + (max_lr - base_lr) × max(0, 1 - |step/step_size - 2×cycle + 1|) × scale_fn
- base_lr
- Lower bound of the learning rate
- max_lr
- Upper bound of the learning rate
- step_size
- Half-cycle length in steps
- cycle
- Current cycle number: floor(1 + step / (2 × step_size))
- scale_fn
- Policy-dependent amplitude scale (1 for triangular, 1/2^(cycle-1) for triangular2, gamma^step for exp_range)
Worked example
With the default values below, Cyclic Learning Rate Calculator returns Learning Rate at Step of 7.7500 × 10⁻⁴. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Base Learning Rateminimum 0
- 0.0001
- Max Learning Rateminimum 0
- 0.001
- Step Size (half-cycle length)accepts 1–1,000,000,000
- 2,000
- Current Stepaccepts 0–1,000,000,000
- 1,500
- Policyone of Triangular, Triangular2 (halves amplitude each cycle), Exp Range (exponential amplitude decay)
- Triangular
- Gamma (exp_range decay constant)accepts 0.9–1
- 0.99994
Results
- Learning Rate at Step
- 7.7500 × 10⁻⁴
- Current Cycle Number
- 1
Step by step
- 1Cycle number: floor(1 + step ÷ (2×step_size))= 1
- 2Triangle position x = |step/step_size − 2×cycle + 1|= 0.2500
- 3Policy scale factor (triangular)= 1.000000
- 4lr = base_lr + (max_lr − base_lr) × max(0, 1−x) × scale= 7.7500e-4
Assumptions and limits
- The result is derived only from the 6 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 4 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
What's the intuition behind cycling the learning rate instead of just decaying it?
Repeatedly increasing the LR partway through training can help the optimizer escape saddle points or sharp minima it might otherwise get stuck in with monotonic decay, at the cost of some added training instability during the high-LR portions of each cycle.
How do I choose step_size?
The original CLR paper suggests setting step_size to somewhere between 2 and 10 times the number of iterations in one epoch — too short a step size doesn't let the model benefit from time at higher or lower LRs; too long slows down the cycling benefit.
What's the difference between triangular2 and exp_range?
Triangular2 halves the oscillation amplitude at fixed intervals (once per full cycle), producing a stair-step reduction in peak LR over time; exp_range shrinks the amplitude continuously and smoothly at every step via gamma^step, giving a smoother decay envelope.
How is CLR related to one-cycle LR?
One-cycle LR can be seen as a single 'half' of a cyclic schedule — it warms up to a peak once and decays once (plus an annihilation phase), rather than repeating the up-down oscillation multiple times across training like classic CLR.