AI engineering glossary · Deep learning core
What is Dying ReLU?
Dying ReLU (also called Dead neuron): A failure in which a ReLU neuron's input becomes negative for all training examples, so it outputs zero and receives zero gradient, and can no longer recover. A too-high learning rate or bad bias values often cause it.
Dying ReLU: an example
After one huge update, a neuron with bias -50 outputs 0 for every input and never changes again.
Related terms
More Deep learning core terms
- ELU
- Embedding layer
- Epoch
- Evaluation mode
- Exploding gradient
- Exponential moving average of weights
- Fan-in and fan-out
- Feedforward neural network
Where you learn it
Dying ReLU is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.