AI engineering glossary · Transformers LLM architecture
What is attention pattern?
Attention pattern: The matrix of attention weights of one head for a given input, showing which token each position attends to. Inspecting patterns can reveal behaviours like attending to the previous token, but they are only part of what a head computes.
Attention pattern: an example
A plotted pattern showing a bright diagonal line one step below the main diagonal indicates a previous-token head.
Related terms
More Transformers LLM architecture terms
- Attention sink
- Attention weights
- Autoregressive generation
- Autoregressive model
- BART
- Beam search
- BERT
- Bidirectional attention
Where you learn it
Attention pattern is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.