AI engineering glossary · Deep learning core
What is gradient norm?
Gradient norm: The overall size of the gradient vector across all parameters, commonly the square root of the sum of squared entries. Tracking it reveals exploding or vanishing gradients and is the quantity used in norm-based clipping.
Gradient norm: an example
A gradient norm that suddenly jumps from 1.2 to 800 is an early warning of instability.
Related terms
More Deep learning core terms
- Group normalisation
- GRU
- He initialisation
- Hidden layer
- Hidden state
- Huber loss
- Input layer
- Instance normalisation
Where you learn it
Gradient norm is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.