Momentum Update Calculator
Calculate the parameter update step produced by gradient descent with momentum.
About Momentum Update Calculator
Classical momentum accumulates a velocity term from past gradients, v ← μ·v − lr·g, and applies it to the parameter: θ ← θ + v. This smooths the optimization trajectory and can accelerate progress through consistent gradient directions. The Nesterov variant computes the gradient with a 'lookahead' — evaluating as if the momentum step had already been taken — which gives θ_new = θ + μ·v_t − lr·g in this simplified form, providing a corrective term that often improves convergence speed and stability slightly over classical momentum.
Formulas
Classical momentum
v_t = mu × v_(t-1) - lr × g; theta_new = theta + v_t
- mu
- Momentum coefficient
- v_(t-1)
- Previous velocity
- lr
- Learning rate
- g
- Gradient
Nesterov momentum
v_t = mu × v_(t-1) - lr × g; theta_new = theta + mu × v_t - lr × g
- mu
- Momentum coefficient
- v_t
- Updated velocity
- lr
- Learning rate
- g
- Gradient
Worked example
With the default values below, Momentum Update Calculator returns Updated Parameter (θ_new) of 1.175000. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Current Parameter (θ)
- 1
- Gradient (g)
- 0.5
- Previous Velocity (v_{t-1})
- 0.2
- Learning Rateminimum 0
- 0.01
- Momentum Coefficient (μ)accepts 0–1
- 0.9
- Use Nesterov Variant
- No
Results
- Updated Parameter (θ_new)
- 1.175000
- Velocity (v_t)
- 0.175000
- Standard Momentum Result
- 1.175000
- Nesterov Momentum Result
- 1.152500
Step by step
- 1v_t = μ×v_{t-1} − lr×g= 0.175000
- 2θ_new = θ + v_t= 1.175000
Assumptions and limits
- The result is derived only from the 6 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 6 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
What's the practical difference between standard and Nesterov momentum?
Standard momentum applies the accumulated velocity blindly; Nesterov momentum effectively 'looks ahead' to where the velocity would take the parameters and adjusts using gradient information from that anticipated position, which tends to correct overshooting slightly earlier and can converge marginally faster in practice.
Why is momentum sometimes visualized as a 'heavy ball rolling downhill'?
The velocity term behaves like physical momentum — it builds up speed when gradients consistently point the same direction (rolling downhill) and resists abrupt direction changes, similar to how a heavy ball's inertia would carry it past small bumps in a valley.
Does high momentum always help?
No — too high a momentum coefficient (close to 1) can cause the optimizer to overshoot minima and oscillate, since the velocity term dominates over the current gradient's corrective signal; μ = 0.9–0.99 is the typical safe range.
How does momentum relate to Adam's first moment?
Adam's first-moment term m is computed with the same exponential-averaging structure as the velocity in classical momentum, but Adam additionally normalizes the update by an adaptive per-parameter scale derived from the second moment (squared gradients).