AI engineering glossary · Transformers LLM architecture
What is mixtral?
Mixtral: A family of open-weight mixture-of-experts language models from Mistral AI, where each layer has several experts and a router selects two of them per token. It demonstrated that a sparse model can match larger dense models at lower per-token compute.
Mixtral: an example
Mixtral 8x7B uses 2 of its 8 experts per token in each layer, so it computes with only a fraction of its total parameters.
Related terms
More Transformers LLM architecture terms
- Mixture of depths
- Mixture of experts
- Model dimension
- Multi-head attention
- Multi-head latent attention
- Multi-query attention
- Multi-token prediction
- Open-weight model
Where you learn it
Mixtral is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.