AI engineering glossary · LLM training alignment
What is long-context fine-tuning?
Long-context fine-tuning (also called Context extension): Extending a model to handle longer inputs by continuing training on long sequences, often with adjusted positional-encoding settings, after pretraining on shorter ones. It is cheaper than pretraining at full length.
Long-context fine-tuning: an example
A model pretrained on 4,096-token windows is further trained on 32,768-token documents.
Related terms
More LLM training alignment terms
- LoRA
- Loss masking
- Loss spike
- Masked language modelling
- Maximal update parametrisation
- Megatron-LM
- Model collapse
- Model FLOPs utilisation
Where you learn it
Long-context fine-tuning is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.