AI engineering glossary · Transformers LLM architecture
What is linear attention?
Linear attention: A family of approximations that replace softmax attention with kernel-style computations, letting the model reorder operations so cost grows linearly with sequence length. Some can be run like recurrent networks with fixed-size memory, but usually with quality trade-offs.
Linear attention: an example
Rather than an n x n score matrix for 100,000 tokens, a running summary of fixed size is updated for each new token.
Related terms
More Transformers LLM architecture terms
Where you learn it
Linear attention is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.