AI engineering glossary · Transformers LLM architecture
What is MTP (multi-token prediction)?
Multi-token prediction (also called MTP): A training approach where the model predicts several future tokens at each position rather than only the next one, using extra prediction heads. It can give a richer training signal and enable faster generation by drafting multiple tokens.
Multi-token prediction: an example
At position t, heads predict tokens t+1, t+2 and t+3 at once during training.
Related terms
More Transformers LLM architecture terms
- Open-weight model
- Padding mask
- Parameter count
- Position interpolation
- Positional encoding
- Post-norm
- Pre-norm
- Prefill and decode
Where you learn it
Multi-token prediction is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.