Calculate the clipped gradient norm given a maximum gradient clipping threshold.
Gradient clipping by norm prevents exploding gradients by rescaling the entire gradient vector whenever its L2 norm exceeds a threshold: if ‖g‖ > max_norm, every component of g is multiplied by max_norm / ‖g‖, so the rescaled gradient's norm equals exactly max_norm while preserving its direction. If ‖g‖ is already at or below max_norm, the gradient is left unchanged. This is distinct from clipping by value, which independently clamps each gradient component to a fixed range regardless of the overall vector norm.
clipped_g = g × (max_norm / ||g||) if ||g|| > max_norm, else g
Common choices range from 0.5 to 5.0 depending on model and task; transformer training often uses 1.0, while some RNN/LSTM training uses higher values like 5.0 — it's usually tuned alongside the learning rate.
Norm clipping rescales the whole gradient vector uniformly, preserving its direction (and thus the optimizer's intended update direction) while only reducing its magnitude — value clipping distorts direction because each component is clamped independently.
No — norm-based clipping scales all components of the gradient vector by the same factor, so the direction is preserved exactly; only the magnitude changes.
It's most valuable in RNNs/LSTMs (prone to exploding gradients through many timesteps) and in early transformer training or when using high learning rates, where occasional large gradient spikes could otherwise destabilize training.