AI engineering glossary · Deep learning core
What is exploding gradient?
Exploding gradient (also called Exploding gradients): A training problem where gradients grow larger and larger while moving backward, causing enormous weight updates, unstable loss and often numerical overflow. It is common in deep or recurrent networks and is typically controlled by clipping and careful initialisation.
Exploding gradient: an example
A loss that jumps from 2.3 to 10^9 to NaN within a few steps often indicates this failure.
Related terms
More Deep learning core terms
- Exponential moving average of weights
- Fan-in and fan-out
- Feedforward neural network
- Flat minima
- Focal loss
- Forward pass
- Fully connected layer
- Gated linear unit
Where you learn it
Exploding gradient is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.