AI engineering glossary · Transformers LLM architecture
What is mixture of experts?
Mixture of experts (also called MoE): An architecture in which each layer's feed-forward block is replaced by several parallel sub-networks called experts, plus a router that sends each token to only a few of them. The model has a large total parameter count but uses only a fraction per token.
Mixture of experts: an example
Mixtral 8x7B has 8 experts per layer and uses 2 per token, so far fewer parameters are active than in total.
Related terms
More Transformers LLM architecture terms
- Model dimension
- Multi-head attention
- Multi-head latent attention
- Multi-query attention
- Multi-token prediction
- Open-weight model
- Padding mask
- Parameter count
Where you learn it
Mixture of experts is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.