AI engineering glossary · Transformers LLM architecture
What is additive attention?
Additive attention (also called Bahdanau attention): An early attention scoring method, introduced for machine translation, where the relevance of an input to the current state is computed by a small feed-forward network over both, passed through tanh. It preceded the dot-product form used in Transformers, which is faster on matrix hardware.
Additive attention: an example
In a GRU translation model, score = v^T tanh(W1*decoder_state + W2*encoder_state) is computed for each source word, then normalised to attention weights.
Related terms
More Transformers LLM architecture terms
- ALiBi
- Attention head
- Attention mechanism
- Attention pattern
- Attention sink
- Attention weights
- Autoregressive generation
- Autoregressive model
Where you learn it
Additive attention is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.