AI engineering glossary · Inference serving optimisation
What is prompt lookup decoding?
Prompt lookup decoding: A simple speculative technique that needs no draft model: it finds the latest words of the output in the prompt and proposes the words that followed them there as guesses. It works well when answers copy from the input, such as summaries or code edits.
Prompt lookup decoding: an example
When rewriting a document with small edits, most output tokens are copied runs from the prompt and are accepted in bulk.
Related terms
- Speculative decoding
- N-gram speculation
- Acceptance rate
- Draft model
More Inference serving optimisation terms
- Pruning
- Quantisation
- Quantisation-aware training
- Queueing delay
- RadixAttention
- Request scheduler
- Reserved capacity
- Roofline model
Where you learn it
Prompt lookup decoding is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.