Exponential Decay Calculator
Calculate the learning rate at a given step under an exponential decay schedule.
Inputs
Multiplicative factor applied every decay_steps.
Interval (in steps) over which one decay_rate multiplication is applied.
Learning Rate at Step
8.1537 × 10⁻⁴
% of Initial LR Remaining
81.54%
Step by step
Exponent: step ÷ decay_steps
5000 ÷ 1000
= 5.0000
lr = lr₀ × decay_rate^(step/decay_steps)
0.001 × 0.96^5.0000
= 8.1537e-4
How it works
Exponential decay reduces the learning rate by a constant multiplicative factor every fixed interval: lr(step) = lr₀ × decay_rate^(step / decay_steps). A decay_rate below 1 (e.g. 0.96) shrinks the learning rate continuously; smaller decay_steps values mean the decay factor is applied more frequently, producing faster overall decay for the same decay_rate. Unlike step decay (which drops the LR in discrete jumps), exponential decay changes smoothly and continuously with every step, though the formula is most often evaluated once per epoch or every N steps in practice.
Formula
lr(step) = lr_0 × decay_rate ^ (step / decay_steps)
- lr_0
- Initial learning rate
- decay_rate
- Multiplicative factor per decay interval (e.g. 0.96)
- step
- Current training step
- decay_steps
- Number of steps per decay application
Frequently Asked Questions
What's a typical decay_rate value?
Values between 0.9 and 0.99 are common for per-epoch decay, while decay_steps set to match one epoch's worth of steps (or a small multiple) is a frequent convention; the specific values depend heavily on total training duration and how aggressively you want the LR to shrink.
How does exponential decay differ from cosine annealing?
Exponential decay never truly reaches zero (asymptotically approaching it) and decays at a constant relative rate throughout training, while cosine annealing decays to an explicit lr_min over a fixed horizon with a characteristic slow-fast-slow curve shape.
What if decay_steps is set to 1?
Then the decay factor is applied at every single step rather than in batches of decay_steps, producing continuous per-step exponential decay — useful for very fine-grained schedules, though it makes the decay_rate value need to be very close to 1 to avoid decaying too fast.
Is exponential decay still commonly used for deep learning?
It's less common than cosine annealing or one-cycle policies for training large modern models, but it remains a simple, well-understood baseline and is still used in some production pipelines and reinforcement learning setups.