AI engineering glossary · Deep learning core
What is siLU?
SiLU (also called Swish, Sigmoid linear unit): The sigmoid linear unit, an activation defined as x * sigmoid(x). It is smooth, non-monotonic near zero and behaves like ReLU for large inputs. It is also known as Swish and appears in many modern vision and language networks.
SiLU: an example
SiLU(2) = 2 * 0.88, about 1.76; SiLU(-2) is about -0.24.
Related terms
More Deep learning core terms
- Skip connection
- Softmax
- Softplus
- Sparse categorical cross-entropy
- Spectral normalisation
- Stochastic depth
- Stochastic weight averaging
- Straight-through estimator
Where you learn it
SiLU is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.