AI engineering glossary · Transformers LLM architecture
What is attention sink?
Attention sink: A phenomenon in which autoregressive Transformers place a large share of attention weight on the first few tokens, regardless of their meaning, giving attention a place to put weight when nothing is relevant. Streaming methods keep these tokens in the cache so generation stays stable.
Attention sink: an example
When a chat runs long and old tokens are evicted, keeping the first 4 tokens in the cache prevents quality from collapsing.
Related terms
More Transformers LLM architecture terms
- Attention weights
- Autoregressive generation
- Autoregressive model
- BART
- Beam search
- BERT
- Bidirectional attention
- BigBird
Where you learn it
Attention sink is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.