AI engineering glossary · Transformers LLM architecture
What is position interpolation?
Position interpolation (also called RoPE scaling, Positional interpolation): A technique to extend a model's usable context by scaling down position indices so that longer inputs fit within the position range seen in training, followed by brief fine-tuning. It avoids the poor behaviour that occurs when rotary positions are pushed past their trained range.
Position interpolation: an example
To take a model trained on 4,096 tokens to 16,384, positions are multiplied by 0.25 so they map back into 0-4,096.
Related terms
More Transformers LLM architecture terms
- Positional encoding
- Post-norm
- Pre-norm
- Prefill and decode
- Prefix language model
- QK-norm
- Quadratic attention cost
- Query, key and value
Where you learn it
Position interpolation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.