AI engineering glossary · Transformers LLM architecture
What is yaRN?
YaRN (also called Yet another RoPE extensioN): A method for extending the context window of models using rotary position embeddings. It rescales different frequency components by different amounts and adjusts attention temperature, so a model can handle much longer inputs with relatively little extra training compared with plain position interpolation.
YaRN: an example
A team applies YaRN to an open model trained on 8K tokens and fine-tunes briefly to serve 64K-token contracts.
Related terms
More Transformers LLM architecture terms
- Activation patching
- Active parameters
- Additive attention
- ALiBi
- Attention head
- Attention mechanism
- Attention pattern
- Attention sink
Where you learn it
YaRN is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.