AI engineering glossary · Transformers LLM architecture
What is reformer?
Reformer: An efficient Transformer that uses locality-sensitive hashing to group similar queries and keys so attention is computed only within buckets, and reversible layers to save memory. It reduces long-sequence cost from quadratic towards near-linear.
Reformer: an example
Tokens whose query vectors hash to the same bucket attend to each other only, instead of to all 64,000 tokens.
Related terms
More Transformers LLM architecture terms
- Relative position encoding
- Residual stream
- Ring attention
- RoBERTa
- Rotary position embedding
- RWKV
- Scaled dot-product attention
- Self-attention
Where you learn it
Reformer is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.