AI engineering glossary · Inference serving optimisation
What is tree attention?
Tree attention: A masking scheme that lets a model score several branching candidate continuations in one forward pass, where each candidate token only attends to its own ancestors. It is used to verify many speculative alternatives at once.
Tree attention: an example
Two candidates for the next word and two follow-ons each are checked in one pass rather than four.
Related terms
More Inference serving optimisation terms
- Triton Inference Server
- Unstructured pruning
- Verification step
- vLLM
- Warp
- Weight-only quantisation
- Zero-point
- Acceptance rate
Where you learn it
Tree attention is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.