Calculate the parameter update step produced by gradient descent with momentum.
Classical momentum accumulates a velocity term from past gradients, v ← μ·v − lr·g, and applies it to the parameter: θ ← θ + v. This smooths the optimization trajectory and can accelerate progress through consistent gradient directions. The Nesterov variant computes the gradient with a 'lookahead' — evaluating as if the momentum step had already been taken — which gives θ_new = θ + μ·v_t − lr·g in this simplified form, providing a corrective term that often improves convergence speed and stability slightly over classical momentum.
Classical momentum
v_t = mu × v_(t-1) - lr × g; theta_new = theta + v_t
Nesterov momentum
v_t = mu × v_(t-1) - lr × g; theta_new = theta + mu × v_t - lr × g
Standard momentum applies the accumulated velocity blindly; Nesterov momentum effectively 'looks ahead' to where the velocity would take the parameters and adjusts using gradient information from that anticipated position, which tends to correct overshooting slightly earlier and can converge marginally faster in practice.
The velocity term behaves like physical momentum — it builds up speed when gradients consistently point the same direction (rolling downhill) and resists abrupt direction changes, similar to how a heavy ball's inertia would carry it past small bumps in a valley.
No — too high a momentum coefficient (close to 1) can cause the optimizer to overshoot minima and oscillate, since the velocity term dominates over the current gradient's corrective signal; μ = 0.9–0.99 is the typical safe range.
Adam's first-moment term m is computed with the same exponential-averaging structure as the velocity in classical momentum, but Adam additionally normalizes the update by an adaptive per-parameter scale derived from the second moment (squared gradients).