Calculate the learning rate at a given step under an exponential decay schedule.
Exponential decay reduces the learning rate by a constant multiplicative factor every fixed interval: lr(step) = lr₀ × decay_rate^(step / decay_steps). A decay_rate below 1 (e.g. 0.96) shrinks the learning rate continuously; smaller decay_steps values mean the decay factor is applied more frequently, producing faster overall decay for the same decay_rate. Unlike step decay (which drops the LR in discrete jumps), exponential decay changes smoothly and continuously with every step, though the formula is most often evaluated once per epoch or every N steps in practice.
lr(step) = lr_0 × decay_rate ^ (step / decay_steps)
Values between 0.9 and 0.99 are common for per-epoch decay, while decay_steps set to match one epoch's worth of steps (or a small multiple) is a frequent convention; the specific values depend heavily on total training duration and how aggressively you want the LR to shrink.
Exponential decay never truly reaches zero (asymptotically approaching it) and decays at a constant relative rate throughout training, while cosine annealing decays to an explicit lr_min over a fixed horizon with a characteristic slow-fast-slow curve shape.
Then the decay factor is applied at every single step rather than in batches of decay_steps, producing continuous per-step exponential decay — useful for very fine-grained schedules, though it makes the decay_rate value need to be very close to 1 to avoid decaying too fast.
It's less common than cosine annealing or one-cycle policies for training large modern models, but it remains a simple, well-understood baseline and is still used in some production pipelines and reinforcement learning setups.