AI engineering glossary · Transformers LLM architecture
What is attention weights?
Attention weights (also called Attention scores): The numbers produced by applying softmax to attention scores, one per pair of positions, that sum to 1 across each row. They show how much each token draws from every other token and can be plotted as a heatmap, though they are only a partial explanation of model behaviour.
Attention weights: an example
For a 6-token sentence, each token has 6 attention weights, for example [0.05, 0.60, 0.10, 0.10, 0.10, 0.05].
Related terms
More Transformers LLM architecture terms
- Autoregressive generation
- Autoregressive model
- BART
- Beam search
- BERT
- Bidirectional attention
- BigBird
- Capacity factor
Where you learn it
Attention weights is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.