AI engineering glossary · Inference serving optimisation
What is speculative decoding?
Speculative decoding (also called Speculative sampling): A way to speed up text generation without changing the output distribution. A small, fast draft model proposes several next tokens and the large model checks them all in one pass, keeping the correct ones. When the draft is often right, several tokens are produced for the price of one large-model step.
Speculative decoding: an example
A 1B draft model proposes 4 tokens; the 70B model accepts 3 of them in one verification pass, so 4 tokens emerge from a single big-model step.
Related terms
More Inference serving optimisation terms
- Spot instance
- Static batching
- Streaming multiprocessor
- Structured pruning
- Symmetric quantisation
- Tail latency
- Target model
- Tensor core
Where you learn it
Speculative decoding is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.