AI engineering glossary · Transformers LLM architecture
What is expert?
Expert: One of the parallel feed-forward sub-networks inside a mixture-of-experts layer. Each token is processed by only a few experts. Experts are not usually specialised in neat human topics, though they can develop tendencies for certain kinds of tokens.
Expert: an example
In a layer with 64 experts, a given token might be processed by experts 7 and 41 only.
Related terms
More Transformers LLM architecture terms
- Expert parallelism
- Fill-in-the-middle
- FlashAttention
- Frequency and presence penalty
- GPT
- Greedy decoding
- Grouped-query attention
- Head dimension
Where you learn it
Expert is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.