AI engineering glossary · Transformers LLM architecture
What is mixture of depths?
Mixture of depths (also called MoD): A technique in which a router decides, per token and per layer, whether the token goes through the block's attention and MLP or skips it via the residual path. Compute is spent only on tokens that need it, with a fixed budget per layer.
Mixture of depths: an example
In each layer only the top 12.5 percent of tokens are processed by the block; the rest pass through unchanged.
Related terms
More Transformers LLM architecture terms
- Mixture of experts
- Model dimension
- Multi-head attention
- Multi-head latent attention
- Multi-query attention
- Multi-token prediction
- Open-weight model
- Padding mask
Where you learn it
Mixture of depths is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.