AI engineering glossary · Transformers LLM architecture
What is expert parallelism?
Expert parallelism: A way of spreading a mixture-of-experts model over many GPUs by placing different experts on different devices, with tokens sent across the network to whichever device hosts the experts chosen for them. It makes very large MoE models fit and run but adds communication cost.
Expert parallelism: an example
With 64 experts on 8 GPUs, each GPU holds 8 experts and receives the tokens routed to them.
Related terms
More Transformers LLM architecture terms
- Fill-in-the-middle
- FlashAttention
- Frequency and presence penalty
- GPT
- Greedy decoding
- Grouped-query attention
- Head dimension
- Hidden states
Where you learn it
Expert parallelism is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.