Calculate the parameter update step produced by the RMSProp optimizer.
RMSProp maintains an exponentially decaying running average of squared gradients, v ← ρ·v + (1−ρ)·g², and divides each gradient step by the square root of that average before scaling by the learning rate: θ ← θ − lr·g/√(v+ε). This normalizes the effective step size per parameter, shrinking updates for parameters with consistently large gradients and growing them for parameters with small gradients — helping training progress evenly across parameters with very different gradient scales, which is especially useful for RNNs.
theta_new = theta - lr × g / sqrt(v_t + epsilon)
Adam's second-moment term v is computed identically to RMSProp's running average of squared gradients; Adam adds a first-moment (momentum-like) term on top and applies bias correction to both moments, making it effectively RMSProp + momentum + bias correction.
ρ = 0.9 is the standard default, giving the running average an effective memory of roughly the last 10 squared-gradient observations.
It was introduced (in an unpublished Coursera lecture by Geoffrey Hinton) to address AdaGrad's problem of an ever-shrinking learning rate — AdaGrad accumulates squared gradients without decay, so its effective learning rate eventually approaches zero, while RMSProp's exponential decay allows it to 'forget' old gradient information and keep adapting.
RMSProp already adapts per-parameter effective step sizes based on gradient history, but the global learning rate is still an important hyperparameter and often benefits from an additional schedule (e.g. decay over training) for best results.