Gradient Clipping Calculator
Calculate the clipped gradient norm given a maximum gradient clipping threshold.
Inputs
Resulting Gradient Norm
5.0000
Clipping Applied?
true
Scale Factor Applied
0.333333
Magnitude Reduction
66.67%
Step by step
Clipping condition: ‖g‖ > max_norm
15 > 5
= True — clipping applied
Scale factor: max_norm ÷ ‖g‖
5 ÷ 15
= 0.333333
Clipped gradient norm: ‖g‖ × scale
15 × 0.3333
= 5.0000
How it works
Gradient clipping by norm prevents exploding gradients by rescaling the entire gradient vector whenever its L2 norm exceeds a threshold: if ‖g‖ > max_norm, every component of g is multiplied by max_norm / ‖g‖, so the rescaled gradient's norm equals exactly max_norm while preserving its direction. If ‖g‖ is already at or below max_norm, the gradient is left unchanged. This is distinct from clipping by value, which independently clamps each gradient component to a fixed range regardless of the overall vector norm.
Formula
clipped_g = g × (max_norm / ||g||) if ||g|| > max_norm, else g
- g
- Gradient vector
- ||g||
- L2 norm of the gradient
- max_norm
- Maximum allowed gradient norm
Frequently Asked Questions
What's a typical max_norm value?
Common choices range from 0.5 to 5.0 depending on model and task; transformer training often uses 1.0, while some RNN/LSTM training uses higher values like 5.0 — it's usually tuned alongside the learning rate.
Why clip by norm instead of by value?
Norm clipping rescales the whole gradient vector uniformly, preserving its direction (and thus the optimizer's intended update direction) while only reducing its magnitude — value clipping distorts direction because each component is clamped independently.
Does gradient clipping change the direction of the update?
No — norm-based clipping scales all components of the gradient vector by the same factor, so the direction is preserved exactly; only the magnitude changes.
When is gradient clipping most needed?
It's most valuable in RNNs/LSTMs (prone to exploding gradients through many timesteps) and in early transformer training or when using high learning rates, where occasional large gradient spikes could otherwise destabilize training.