AI engineering glossary · Transformers LLM architecture
What is attention mechanism?
Attention mechanism (also called Attention): A way for a model to decide which parts of its input matter most for the current step. It scores how relevant each input item is, turns the scores into weights that sum to 1, and takes a weighted average of the items. This lets the model focus on distant but relevant information.
Attention mechanism: an example
When translating 'The cat sat on the mat' word by word, the model puts most attention weight on 'cat' while producing the matching verb form.
Related terms
More Transformers LLM architecture terms
- Attention pattern
- Attention sink
- Attention weights
- Autoregressive generation
- Autoregressive model
- BART
- Beam search
- BERT
Where you learn it
Attention mechanism is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.