AI engineering glossary · Transformers LLM architecture
What is shared expert?
Shared expert: An expert in a mixture-of-experts layer that every token always passes through, alongside the routed experts. It captures common knowledge so that the routed experts can specialise. DeepSeek's MoE models use this idea.
Shared expert: an example
A layer has 1 shared expert plus 64 routed experts, of which 6 are chosen per token.
Related terms
More Transformers LLM architecture terms
- Sinusoidal positional encoding
- Sliding window attention
- Sparse attention
- Sparse autoencoder
- State space model
- Switch Transformer
- T5
- Temperature
Where you learn it
Shared expert is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.