AI engineering glossary · Transformers LLM architecture
What is sparse attention?
Sparse attention: Attention in which each token attends to only a chosen subset of positions, such as a local window, a few global tokens or a strided pattern, instead of all of them. It cuts the cost of long sequences at the price of some flexibility.
Sparse attention: an example
A pattern where each token sees its 512 neighbours plus 4 global tokens replaces a full 16,000 x 16,000 score matrix.
Related terms
More Transformers LLM architecture terms
- Sparse autoencoder
- State space model
- Switch Transformer
- T5
- Temperature
- Test-time compute
- Top-k expert routing
- Top-k sampling
Where you learn it
Sparse attention is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.