SGD Update Calculator
Calculate the parameter update step produced by stochastic gradient descent.
About SGD Update Calculator
Vanilla stochastic gradient descent updates each parameter by stepping directly against its gradient, scaled by the learning rate: θ ← θ − lr·g. With momentum added, an exponentially-decaying velocity term accumulates gradient history: v ← μ·v − lr·g, and the parameter update uses the velocity instead of the raw gradient: θ ← θ + v. Momentum smooths out noisy per-step gradients and accelerates progress along consistent gradient directions, at the cost of an extra hyperparameter (μ, typically 0.9) and one extra state variable per parameter.
Formulas
Vanilla SGD
theta_new = theta - lr × g
- theta
- Current parameter value
- lr
- Learning rate
- g
- Gradient
SGD with momentum
v_t = mu × v_(t-1) - lr × g; theta_new = theta + v_t
- mu
- Momentum coefficient
- v_(t-1)
- Previous velocity
- lr
- Learning rate
- g
- Gradient
Worked example
With the default values below, SGD Update Calculator returns Updated Parameter (θ_new) of 0.995000. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Current Parameter (θ)
- 1
- Gradient (g)
- 0.5
- Learning Rateminimum 0
- 0.01
- Use Momentum
- No
- Momentum Coefficient (μ)accepts 0–1
- 0.9
- Previous Velocity (v_{t-1})
- 0
Results
- Updated Parameter (θ_new)
- 0.995000
- Update Applied
- 0.005000
Step by step
- 1θ_new = θ − lr × g= 0.995000
Assumptions and limits
- The result is derived only from the 6 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 6 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
Why add momentum to SGD?
Momentum accumulates a running average of past gradients, which smooths out noise from mini-batch sampling and helps the optimizer power through shallow local curvature or saddle points, typically leading to faster and more stable convergence than vanilla SGD.
What's a typical momentum coefficient?
μ = 0.9 is the most common default, meaning the velocity effectively averages over roughly the last 10 gradients; higher values (e.g. 0.99) give a longer memory and smoother but slower-to-adapt trajectories.
How does SGD with momentum compare to Adam?
SGD+momentum uses one global learning rate for all parameters and no per-parameter adaptive scaling, while Adam adapts the effective learning rate per parameter based on gradient magnitude history — SGD+momentum with a well-tuned learning rate schedule is still competitive (and sometimes generalizes better) on many vision tasks, while Adam is more robust to hyperparameter choice out of the box.
What is the difference between this and Nesterov momentum?
Standard momentum computes the gradient at the current position before applying velocity; Nesterov momentum computes the gradient at a 'lookahead' position (θ + μ·v) first, giving a slight correction that often converges faster — see the Momentum Update Calculator for the Nesterov variant.