AI engineering glossary · Deep learning core
What is saturation?
Saturation (also called Saturating activation): A state in which an activation function is so far into its flat region that changing the input barely changes the output, so its gradient is nearly zero. Saturated units learn very slowly, a key reason sigmoid and tanh hurt deep networks.
Saturation: an example
Sigmoid(10) is about 0.99995, and its slope there is roughly 0.00005, so updates almost stop.
Related terms
More Deep learning core terms
- SELU
- SiLU
- Skip connection
- Softmax
- Softplus
- Sparse categorical cross-entropy
- Spectral normalisation
- Stochastic depth
Where you learn it
Saturation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.