AI engineering glossary · Deep learning core
What is GELU?
GELU (also called Gaussian error linear unit): The Gaussian error linear unit, a smooth activation that multiplies the input by the probability that a standard normal variable is below it: x * Phi(x). Unlike ReLU it curves gently near zero, and it is widely used in transformer models such as BERT and GPT.
GELU: an example
GELU(1) is about 0.84, while GELU(-1) is about -0.16.
Related terms
More Deep learning core terms
- Gradient checkpointing
- Gradient norm
- Group normalisation
- GRU
- He initialisation
- Hidden layer
- Hidden state
- Huber loss
Where you learn it
GELU is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.