AI engineering glossary · Transformers LLM architecture
What is Switch Transformer?
Switch Transformer: A mixture-of-experts Transformer from Google that simplifies routing by sending each token to just one expert (top-1). It showed that sparse models with very many parameters can train efficiently and introduced a capacity limit and an auxiliary balancing loss.
Switch Transformer: an example
In a Switch layer with 128 experts, each token is handled by exactly one expert feed-forward network.
Related terms
More Transformers LLM architecture terms
- T5
- Temperature
- Test-time compute
- Top-k expert routing
- Top-k sampling
- Top-p sampling
- Transformer
- Transformer block
Where you learn it
Switch Transformer is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.