AI engineering glossary · Transformers LLM architecture
What is bidirectional attention?
Bidirectional attention: Attention in which every token can see all other tokens, both earlier and later, with no causal mask. It gives richer representations for understanding tasks such as classification and tagging, but cannot be used directly for left-to-right text generation.
Bidirectional attention: an example
In BERT, the word 'bank' in 'river bank' is encoded using words both before and after it.
Related terms
More Transformers LLM architecture terms
- BigBird
- Capacity factor
- Causal masking
- Constrained decoding
- Context window
- Contrastive decoding
- Cross-attention
- Decoder
Where you learn it
Bidirectional attention is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.