Calculate the learning rate at a given step under a cyclic learning rate schedule.
Cyclic learning rate (CLR) policies oscillate the learning rate between a base_lr and max_lr repeatedly throughout training instead of monotonically decaying. The 'triangular' policy is a simple linear ramp up and down every 2×step_size steps. 'Triangular2' halves the amplitude (max_lr − base_lr) after every full cycle, giving progressively smaller oscillations. 'Exp_range' instead shrinks the amplitude exponentially with each step via a decay constant gamma (close to but below 1). Cycling the learning rate lets training periodically revisit higher LRs, which can help escape sharp local minima and reduces the need to manually tune a single optimal learning rate.
lr = base_lr + (max_lr - base_lr) × max(0, 1 - |step/step_size - 2×cycle + 1|) × scale_fn
Repeatedly increasing the LR partway through training can help the optimizer escape saddle points or sharp minima it might otherwise get stuck in with monotonic decay, at the cost of some added training instability during the high-LR portions of each cycle.
The original CLR paper suggests setting step_size to somewhere between 2 and 10 times the number of iterations in one epoch — too short a step size doesn't let the model benefit from time at higher or lower LRs; too long slows down the cycling benefit.
Triangular2 halves the oscillation amplitude at fixed intervals (once per full cycle), producing a stair-step reduction in peak LR over time; exp_range shrinks the amplitude continuously and smoothly at every step via gamma^step, giving a smoother decay envelope.
One-cycle LR can be seen as a single 'half' of a cyclic schedule — it warms up to a peak once and decays once (plus an annihilation phase), rather than repeating the up-down oscillation multiple times across training like classic CLR.