Calculate the parameter update step produced by stochastic gradient descent.
Vanilla stochastic gradient descent updates each parameter by stepping directly against its gradient, scaled by the learning rate: θ ← θ − lr·g. With momentum added, an exponentially-decaying velocity term accumulates gradient history: v ← μ·v − lr·g, and the parameter update uses the velocity instead of the raw gradient: θ ← θ + v. Momentum smooths out noisy per-step gradients and accelerates progress along consistent gradient directions, at the cost of an extra hyperparameter (μ, typically 0.9) and one extra state variable per parameter.
Vanilla SGD
theta_new = theta - lr × g
SGD with momentum
v_t = mu × v_(t-1) - lr × g; theta_new = theta + v_t
Momentum accumulates a running average of past gradients, which smooths out noise from mini-batch sampling and helps the optimizer power through shallow local curvature or saddle points, typically leading to faster and more stable convergence than vanilla SGD.
μ = 0.9 is the most common default, meaning the velocity effectively averages over roughly the last 10 gradients; higher values (e.g. 0.99) give a longer memory and smoother but slower-to-adapt trajectories.
SGD+momentum uses one global learning rate for all parameters and no per-parameter adaptive scaling, while Adam adapts the effective learning rate per parameter based on gradient magnitude history — SGD+momentum with a well-tuned learning rate schedule is still competitive (and sometimes generalizes better) on many vision tasks, while Adam is more robust to hyperparameter choice out of the box.
Standard momentum computes the gradient at the current position before applying velocity; Nesterov momentum computes the gradient at a 'lookahead' position (θ + μ·v) first, giving a slight correction that often converges faster — see the Momentum Update Calculator for the Nesterov variant.