AI engineering glossary · Transformers LLM architecture
What is top-k sampling?
Top-k sampling: A sampling method that keeps only the k most probable next tokens, renormalises their probabilities, and picks one at random. It removes the unlikely long tail that can cause nonsense, but a fixed k is too loose when the model is sure and too tight when it is not.
Top-k sampling: an example
With k = 40, only the 40 highest-probability tokens are candidates; all others are excluded.
Related terms
More Transformers LLM architecture terms
- Top-p sampling
- Transformer
- Transformer block
- Transformer MLP block
- Transformer-XL
- Typical sampling
- Universal Transformer
- YaRN
Where you learn it
Top-k sampling is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.