AI engineering glossary · Deep learning core
What is Leaky ReLU?
Leaky ReLU: A variant of ReLU that lets a small fraction of negative inputs through instead of zeroing them: f(x) = x if x > 0, otherwise a*x with a small constant such as 0.01. This keeps a gradient flowing for negative values and reduces neurons getting stuck.
Leaky ReLU: an example
With slope 0.01, an input of -5 outputs -0.05, while ReLU would output 0.
Related terms
More Deep learning core terms
- Learning rate finder
- LeCun initialisation
- Linear scaling rule
- Lion optimiser
- Log-sum-exp trick
- Logits
- Loss scaling
- Lottery ticket hypothesis
Where you learn it
Leaky ReLU is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.