RMSProp Update Calculator
Calculate the parameter update step produced by the RMSProp optimizer.
Inputs
Updated Parameter (θ_new)
0.99852558
Update Applied
0.00147442
Running Average (v_t)
0.115000
Step by step
v_t = ρ×v_{t-1} + (1−ρ)×g²
0.9×0.1 + 0.100×0.5²
= 0.115000
Update = lr × g ÷ √(v_t + ε)
0.001 × 0.5 ÷ √(0.115000 + 1e-8)
= 0.00147442
θ_new = θ − update
1 − 0.00147442
= 0.99852558
How it works
RMSProp maintains an exponentially decaying running average of squared gradients, v ← ρ·v + (1−ρ)·g², and divides each gradient step by the square root of that average before scaling by the learning rate: θ ← θ − lr·g/√(v+ε). This normalizes the effective step size per parameter, shrinking updates for parameters with consistently large gradients and growing them for parameters with small gradients — helping training progress evenly across parameters with very different gradient scales, which is especially useful for RNNs.
Formula
theta_new = theta - lr × g / sqrt(v_t + epsilon)
- v_t
- Running average of squared gradients: rho × v_(t-1) + (1-rho) × g^2
- lr
- Learning rate
- g
- Current gradient
- epsilon
- Small constant for numerical stability
Frequently Asked Questions
How does RMSProp relate to Adam?
Adam's second-moment term v is computed identically to RMSProp's running average of squared gradients; Adam adds a first-moment (momentum-like) term on top and applies bias correction to both moments, making it effectively RMSProp + momentum + bias correction.
What's a typical decay rate for RMSProp?
ρ = 0.9 is the standard default, giving the running average an effective memory of roughly the last 10 squared-gradient observations.
Why was RMSProp originally developed?
It was introduced (in an unpublished Coursera lecture by Geoffrey Hinton) to address AdaGrad's problem of an ever-shrinking learning rate — AdaGrad accumulates squared gradients without decay, so its effective learning rate eventually approaches zero, while RMSProp's exponential decay allows it to 'forget' old gradient information and keep adapting.
Does RMSProp need a learning rate schedule?
RMSProp already adapts per-parameter effective step sizes based on gradient history, but the global learning rate is still an important hyperparameter and often benefits from an additional schedule (e.g. decay over training) for best results.