AI engineering glossary · Transformers LLM architecture
What is flashAttention?
FlashAttention: An exact attention algorithm that reorganises the computation into tiles that fit in fast on-chip GPU memory, avoiding writing the huge attention score matrix to slower memory. The output is the same as standard attention but runs faster and uses much less memory.
FlashAttention: an example
Switching an attention implementation to FlashAttention lets a 32,000-token sequence train without running out of GPU memory.
Related terms
More Transformers LLM architecture terms
- Frequency and presence penalty
- GPT
- Greedy decoding
- Grouped-query attention
- Head dimension
- Hidden states
- Induction head
- Instruct model
Where you learn it
FlashAttention is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.