AI engineering glossary · Transformers LLM architecture
What is cross-attention?
Cross-attention (also called Encoder-decoder attention): Attention in which queries come from one sequence and keys and values come from another. In an encoder-decoder model, decoder positions use it to read the encoder's representation of the input. It is also used to condition image generators on text and to inject retrieved passages.
Cross-attention: an example
In English-to-Hindi translation, each Hindi word being generated queries the encoded English sentence through cross-attention.
Related terms
More Transformers LLM architecture terms
Where you learn it
Cross-attention is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.