AI engineering glossary · Deep learning core
What is GLU (gated linear unit)?
Gated linear unit (also called GLU): A layer component that multiplies one linear projection of the input by a sigmoid-gated second projection: (xW + b) * sigmoid(xV + c). The gate learns to let information through selectively and is the basis of several transformer feed-forward variants.
Gated linear unit: an example
With projections giving 4.0 and a gate value of 0.25, the unit outputs 1.0.
Related terms
More Deep learning core terms
- Gating mechanism
- GELU
- Gradient checkpointing
- Gradient norm
- Group normalisation
- GRU
- He initialisation
- Hidden layer
Where you learn it
Gated linear unit is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.