AI engineering glossary · Calculus optimisation
What is gradient clipping?
Gradient clipping (also called Gradient norm clipping): A safeguard that limits the size of gradients before an update, either by capping each component or by rescaling the whole gradient when its norm exceeds a threshold. It prevents sudden huge steps that could destabilise training, especially in recurrent and transformer models.
Gradient clipping: an example
If the gradient norm is 50 and the threshold is 1, the gradient is multiplied by 1/50 so its new norm is 1: torch.nn.utils.clip_grad_norm_(params, 1.0).
Related terms
More Calculus optimisation terms
- Gradient descent
- Hessian matrix
- Integral
- Jacobian matrix
- Jacobian-vector product
- KKT conditions
- L-BFGS
- L-smoothness
Where you learn it
Gradient clipping is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.