AI engineering glossary · Deep learning core
What is swiGLU?
SwiGLU: A gated feed-forward variant that uses SiLU (Swish) as the gate's activation, multiplying it by a second linear projection. It is used in the feed-forward blocks of several recent large language models as a stronger alternative to plain ReLU or GELU layers.
SwiGLU: an example
A block computes SiLU(xW) * (xV) and then projects the result back down with a third matrix.
Related terms
More Deep learning core terms
- Symmetry breaking
- Tanh
- Teacher forcing
- Training loop
- Universal approximation theorem
- Validation loss
- Vanishing gradient
- Weight decay
Where you learn it
SwiGLU is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.