Gradient Clipping Calculator
Calculate the clipped gradient norm given a maximum gradient clipping threshold.
About Gradient Clipping Calculator
Gradient clipping by norm prevents exploding gradients by rescaling the entire gradient vector whenever its L2 norm exceeds a threshold: if ‖g‖ > max_norm, every component of g is multiplied by max_norm / ‖g‖, so the rescaled gradient's norm equals exactly max_norm while preserving its direction. If ‖g‖ is already at or below max_norm, the gradient is left unchanged. This is distinct from clipping by value, which independently clamps each gradient component to a fixed range regardless of the overall vector norm.
Formula
clipped_g = g × (max_norm / ||g||) if ||g|| > max_norm, else g
- g
- Gradient vector
- ||g||
- L2 norm of the gradient
- max_norm
- Maximum allowed gradient norm
Worked example
With the default values below, Gradient Clipping Calculator returns Resulting Gradient Norm of 5.0000. Change any input above and every figure here is recalculated in your browser.
Inputs used
- Current Gradient Norm (‖g‖)accepts 0–1,000,000,000
- 15
- Max Norm Thresholdaccepts 0–1,000,000,000
- 5
Results
- Clipping Applied?
- true
- Resulting Gradient Norm
- 5.0000
- Scale Factor Applied
- 0.333333
- Magnitude Reduction
- 66.67%
Step by step
- 1Clipping condition: ‖g‖ > max_norm= True — clipping applied
- 2Scale factor: max_norm ÷ ‖g‖= 0.333333
- 3Clipped gradient norm: ‖g‖ × scale= 5.0000
Assumptions and limits
- The result is derived only from the 2 inputs listed above. Anything not among them is not part of the calculation.
- In particular, this calculator does not take inflation, tax, fees or charges or compounding frequency into account.
- Figures are rounded for display to 6 decimal places, so totals built from several rounded parts can differ slightly from the unrounded arithmetic.
How these calculators are built and tested is described on the Methodology page. If a figure looks wrong, please report it.
Frequently Asked Questions
What's a typical max_norm value?
Common choices range from 0.5 to 5.0 depending on model and task; transformer training often uses 1.0, while some RNN/LSTM training uses higher values like 5.0 — it's usually tuned alongside the learning rate.
Why clip by norm instead of by value?
Norm clipping rescales the whole gradient vector uniformly, preserving its direction (and thus the optimizer's intended update direction) while only reducing its magnitude — value clipping distorts direction because each component is clamped independently.
Does gradient clipping change the direction of the update?
No — norm-based clipping scales all components of the gradient vector by the same factor, so the direction is preserved exactly; only the magnitude changes.
When is gradient clipping most needed?
It's most valuable in RNNs/LSTMs (prone to exploding gradients through many timesteps) and in early transformer training or when using high learning rates, where occasional large gradient spikes could otherwise destabilize training.