AI engineering glossary · Deep learning core
What is he initialisation?
He initialisation (also called Kaiming initialisation): A scheme that draws initial weights with variance 2 / fan_in, compensating for ReLU zeroing about half its inputs. It keeps activations from shrinking layer after layer in deep ReLU networks.
He initialisation: an example
For a layer with 512 inputs, weights have standard deviation sqrt(2/512) = 0.0625.
Related terms
More Deep learning core terms
- Hidden layer
- Hidden state
- Huber loss
- Input layer
- Instance normalisation
- Iteration
- Label smoothing
- LAMB optimiser
Where you learn it
He initialisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.