AI engineering glossary · LLM training alignment
What is continued pretraining?
Continued pretraining (also called Continual pretraining): Further training of an existing pretrained model on new unlabelled text with the same next-token objective, typically to add a domain or language. It differs from fine-tuning in using raw text rather than labelled examples.
Continued pretraining: an example
Continuing pretraining a general model on 20 billion tokens of Indian legal text before instruction tuning.
Related terms
More LLM training alignment terms
- Critical batch size
- Data curation
- Data mixture
- Debate (AI safety)
- DeepSpeed
- Direct preference optimisation
- Domain-adaptive pretraining
- DoRA
Where you learn it
Continued pretraining is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.