Cyclic Learning Rate Calculator
Calculate the learning rate at a given step under a cyclic learning rate schedule.
Inputs
Learning Rate at Step
7.7500 × 10⁻⁴
Current Cycle Number
1
Step by step
Cycle number: floor(1 + step ÷ (2×step_size))
floor(1 + 1500 ÷ (2×2000))
= 1
Triangle position x = |step/step_size − 2×cycle + 1|
= 0.2500
Policy scale factor (triangular)
= 1.000000
lr = base_lr + (max_lr − base_lr) × max(0, 1−x) × scale
0.0001 + (0.001−0.0001) × 0.7500 × 1.0000
= 7.7500e-4
How it works
Cyclic learning rate (CLR) policies oscillate the learning rate between a base_lr and max_lr repeatedly throughout training instead of monotonically decaying. The 'triangular' policy is a simple linear ramp up and down every 2×step_size steps. 'Triangular2' halves the amplitude (max_lr − base_lr) after every full cycle, giving progressively smaller oscillations. 'Exp_range' instead shrinks the amplitude exponentially with each step via a decay constant gamma (close to but below 1). Cycling the learning rate lets training periodically revisit higher LRs, which can help escape sharp local minima and reduces the need to manually tune a single optimal learning rate.
Formula
lr = base_lr + (max_lr - base_lr) × max(0, 1 - |step/step_size - 2×cycle + 1|) × scale_fn
- base_lr
- Lower bound of the learning rate
- max_lr
- Upper bound of the learning rate
- step_size
- Half-cycle length in steps
- cycle
- Current cycle number: floor(1 + step / (2 × step_size))
- scale_fn
- Policy-dependent amplitude scale (1 for triangular, 1/2^(cycle-1) for triangular2, gamma^step for exp_range)
Frequently Asked Questions
What's the intuition behind cycling the learning rate instead of just decaying it?
Repeatedly increasing the LR partway through training can help the optimizer escape saddle points or sharp minima it might otherwise get stuck in with monotonic decay, at the cost of some added training instability during the high-LR portions of each cycle.
How do I choose step_size?
The original CLR paper suggests setting step_size to somewhere between 2 and 10 times the number of iterations in one epoch — too short a step size doesn't let the model benefit from time at higher or lower LRs; too long slows down the cycling benefit.
What's the difference between triangular2 and exp_range?
Triangular2 halves the oscillation amplitude at fixed intervals (once per full cycle), producing a stair-step reduction in peak LR over time; exp_range shrinks the amplitude continuously and smoothly at every step via gamma^step, giving a smoother decay envelope.
How is CLR related to one-cycle LR?
One-cycle LR can be seen as a single 'half' of a cyclic schedule — it warms up to a peak once and decays once (plus an annihilation phase), rather than repeating the up-down oscillation multiple times across training like classic CLR.