AI engineering glossary · Transformers LLM architecture
What is relative position encoding?
Relative position encoding: A family of position schemes in which attention depends on the distance between two tokens rather than their absolute indices. This generalises better to different lengths and captures patterns like 'the word two positions back'. RoPE, ALiBi and T5's bias buckets are examples.
Relative position encoding: an example
Whether a verb is 2 tokens after its subject matters the same way at positions 5 and 500.
Related terms
More Transformers LLM architecture terms
- Residual stream
- Ring attention
- RoBERTa
- Rotary position embedding
- RWKV
- Scaled dot-product attention
- Self-attention
- Sequence length
Where you learn it
Relative position encoding is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.