RMSProp Update Calculator
Calculate the parameter update step produced by the RMSProp optimizer.
About RMSProp Update Calculator
RMSProp maintains an exponentially decaying running average of squared gradients, v ← ρ·v + (1−ρ)·g², and divides each gradient step by the square root of that average before scaling by the learning rate: θ ← θ − lr·g/√(v+ε). This normalizes the effective step size per parameter, shrinking updates for parameters with consistently large gradients and growing them for parameters with small gradients — helping training progress evenly across parameters with very different gradient scales, which is especially useful for RNNs.
Formula
theta_new = theta - lr × g / sqrt(v_t + epsilon)
- v_t
- Running average of squared gradients: rho × v_(t-1) + (1-rho) × g^2
- lr
- Learning rate
- g
- Current gradient
- epsilon
- Small constant for numerical stability
Worked example
With the default values below, RMSProp Update Calculator returns Updated Parameter (θ_new) of 0.99852558. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Current Parameter (θ)
- 1
- Gradient (g)
- 0.5
- Previous Running Average (v_{t-1})minimum 0
- 0.1
- Learning Rateminimum 0
- 0.001
- Decay Rate (ρ)accepts 0–1
- 0.9
- ε (stability constant)minimum 0
- 0
Results
- Updated Parameter (θ_new)
- 0.99852558
- Update Applied
- 0.00147442
- Running Average (v_t)
- 0.115000
Step by step
- 1v_t = ρ×v_{t-1} + (1−ρ)×g²= 0.115000
- 2Update = lr × g ÷ √(v_t + ε)= 0.00147442
- 3θ_new = θ − update= 0.99852558
Assumptions and limits
- The result is derived only from the 6 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 8 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
How does RMSProp relate to Adam?
Adam's second-moment term v is computed identically to RMSProp's running average of squared gradients; Adam adds a first-moment (momentum-like) term on top and applies bias correction to both moments, making it effectively RMSProp + momentum + bias correction.
What's a typical decay rate for RMSProp?
ρ = 0.9 is the standard default, giving the running average an effective memory of roughly the last 10 squared-gradient observations.
Why was RMSProp originally developed?
It was introduced (in an unpublished Coursera lecture by Geoffrey Hinton) to address AdaGrad's problem of an ever-shrinking learning rate — AdaGrad accumulates squared gradients without decay, so its effective learning rate eventually approaches zero, while RMSProp's exponential decay allows it to 'forget' old gradient information and keep adapting.
Does RMSProp need a learning rate schedule?
RMSProp already adapts per-parameter effective step sizes based on gradient history, but the global learning rate is still an important hyperparameter and often benefits from an additional schedule (e.g. decay over training) for best results.