AI engineering glossary · Transformers LLM architecture
What is rotary position embedding?
Rotary position embedding (also called RoPE): A method that encodes position by rotating the query and key vectors by an angle proportional to their position, in pairs of dimensions. The dot product between a query and a key then depends on their relative distance. It is widely used in open models such as Llama.
Rotary position embedding: an example
A query at position 10 and a key at position 7 are rotated by different angles, and their dot product reflects the distance of 3.
Related terms
More Transformers LLM architecture terms
- RWKV
- Scaled dot-product attention
- Self-attention
- Sequence length
- Sequence-to-sequence model
- Shared expert
- Sinusoidal positional encoding
- Sliding window attention
Where you learn it
Rotary position embedding is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.