AI engineering glossary · Transformers LLM architecture
What is sliding window attention?
Sliding window attention (also called Local attention): A pattern in which each token attends only to a fixed number of most recent tokens instead of the whole sequence. Cost grows linearly with length, and information can still travel further because stacked layers extend the effective reach.
Sliding window attention: an example
With a window of 4,096, token 10,000 attends to tokens 5,905 through 10,000 only.
Related terms
More Transformers LLM architecture terms
- Sparse attention
- Sparse autoencoder
- State space model
- Switch Transformer
- T5
- Temperature
- Test-time compute
- Top-k expert routing
Where you learn it
Sliding window attention is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.