Momentum Update Calculator
Calculate the parameter update step produced by gradient descent with momentum.
Inputs
Updated Parameter (θ_new)
1.175000
Velocity (v_t)
0.175000
Standard Momentum Result
1.175000
Nesterov Momentum Result
1.152500
Step by step
v_t = μ×v_{t-1} − lr×g
0.9×0.2 − 0.01×0.5
= 0.175000
θ_new = θ + v_t
1 + 0.175000
= 1.175000
How it works
Classical momentum accumulates a velocity term from past gradients, v ← μ·v − lr·g, and applies it to the parameter: θ ← θ + v. This smooths the optimization trajectory and can accelerate progress through consistent gradient directions. The Nesterov variant computes the gradient with a 'lookahead' — evaluating as if the momentum step had already been taken — which gives θ_new = θ + μ·v_t − lr·g in this simplified form, providing a corrective term that often improves convergence speed and stability slightly over classical momentum.
Formulas
Classical momentum
v_t = mu × v_(t-1) - lr × g; theta_new = theta + v_t
- mu
- Momentum coefficient
- v_(t-1)
- Previous velocity
- lr
- Learning rate
- g
- Gradient
Nesterov momentum
v_t = mu × v_(t-1) - lr × g; theta_new = theta + mu × v_t - lr × g
- mu
- Momentum coefficient
- v_t
- Updated velocity
- lr
- Learning rate
- g
- Gradient
Frequently Asked Questions
What's the practical difference between standard and Nesterov momentum?
Standard momentum applies the accumulated velocity blindly; Nesterov momentum effectively 'looks ahead' to where the velocity would take the parameters and adjusts using gradient information from that anticipated position, which tends to correct overshooting slightly earlier and can converge marginally faster in practice.
Why is momentum sometimes visualized as a 'heavy ball rolling downhill'?
The velocity term behaves like physical momentum — it builds up speed when gradients consistently point the same direction (rolling downhill) and resists abrupt direction changes, similar to how a heavy ball's inertia would carry it past small bumps in a valley.
Does high momentum always help?
No — too high a momentum coefficient (close to 1) can cause the optimizer to overshoot minima and oscillate, since the velocity term dominates over the current gradient's corrective signal; μ = 0.9–0.99 is the typical safe range.
How does momentum relate to Adam's first moment?
Adam's first-moment term m is computed with the same exponential-averaging structure as the velocity in classical momentum, but Adam additionally normalizes the update by an adaptive per-parameter scale derived from the second moment (squared gradients).