AI engineering glossary · Transformers LLM architecture
What is quadratic attention cost?
Quadratic attention cost (also called O(n^2) attention): The fact that standard self-attention compares every token with every other, so compute and memory for the score matrix grow with the square of sequence length. Doubling the input length roughly quadruples attention cost, which is the main obstacle to very long contexts.
Quadratic attention cost: an example
A 1,000-token input produces 1 million scores per head; a 10,000-token input produces 100 million.
Related terms
More Transformers LLM architecture terms
- Query, key and value
- Reformer
- Relative position encoding
- Residual stream
- Ring attention
- RoBERTa
- Rotary position embedding
- RWKV
Where you learn it
Quadratic attention cost is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.