AI engineering glossary · Transformers LLM architecture
What is ring attention?
Ring attention: A method for training and running attention over very long sequences by splitting the sequence across many devices and passing blocks of keys and values around a ring of devices, overlapping communication with computation. Memory per device stays small while context grows with device count.
Ring attention: an example
A 1-million-token sequence is split over 64 GPUs, each holding about 16,000 tokens and receiving key-value blocks from its neighbour.
Related terms
More Transformers LLM architecture terms
- RoBERTa
- Rotary position embedding
- RWKV
- Scaled dot-product attention
- Self-attention
- Sequence length
- Sequence-to-sequence model
- Shared expert
Where you learn it
Ring attention is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.