AI engineering glossary · Deep learning core
What is vanishing gradient?
Vanishing gradient (also called Vanishing gradients): A training problem in which gradients shrink toward zero as they are propagated back through many layers, so early layers barely learn. It is caused by repeatedly multiplying small numbers, for example slopes of saturating activations below 1.
Vanishing gradient: an example
Multiplying twenty factors of 0.25, the maximum slope of sigmoid, gives about 10^-12, so early-layer weights effectively freeze.
Related terms
More Deep learning core terms
- Weight decay
- Weight initialisation
- Weight normalisation
- Weight sharing
- Weight tying
- Xavier initialisation
- Activation function
- Adafactor
Where you learn it
Vanishing gradient is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.