SGD Update Calculator
Calculate the parameter update step produced by stochastic gradient descent.
Inputs
Updated Parameter (θ_new)
0.995000
Update Applied
0.005000
Step by step
θ_new = θ − lr × g
1 − 0.01 × 0.5
= 0.995000
How it works
Vanilla stochastic gradient descent updates each parameter by stepping directly against its gradient, scaled by the learning rate: θ ← θ − lr·g. With momentum added, an exponentially-decaying velocity term accumulates gradient history: v ← μ·v − lr·g, and the parameter update uses the velocity instead of the raw gradient: θ ← θ + v. Momentum smooths out noisy per-step gradients and accelerates progress along consistent gradient directions, at the cost of an extra hyperparameter (μ, typically 0.9) and one extra state variable per parameter.
Formulas
Vanilla SGD
theta_new = theta - lr × g
- theta
- Current parameter value
- lr
- Learning rate
- g
- Gradient
SGD with momentum
v_t = mu × v_(t-1) - lr × g; theta_new = theta + v_t
- mu
- Momentum coefficient
- v_(t-1)
- Previous velocity
- lr
- Learning rate
- g
- Gradient
Frequently Asked Questions
Why add momentum to SGD?
Momentum accumulates a running average of past gradients, which smooths out noise from mini-batch sampling and helps the optimizer power through shallow local curvature or saddle points, typically leading to faster and more stable convergence than vanilla SGD.
What's a typical momentum coefficient?
μ = 0.9 is the most common default, meaning the velocity effectively averages over roughly the last 10 gradients; higher values (e.g. 0.99) give a longer memory and smoother but slower-to-adapt trajectories.
How does SGD with momentum compare to Adam?
SGD+momentum uses one global learning rate for all parameters and no per-parameter adaptive scaling, while Adam adapts the effective learning rate per parameter based on gradient magnitude history — SGD+momentum with a well-tuned learning rate schedule is still competitive (and sometimes generalizes better) on many vision tasks, while Adam is more robust to hyperparameter choice out of the box.
What is the difference between this and Nesterov momentum?
Standard momentum computes the gradient at the current position before applying velocity; Nesterov momentum computes the gradient at a 'lookahead' position (θ + μ·v) first, giving a slight correction that often converges faster — see the Momentum Update Calculator for the Nesterov variant.