AI engineering glossary · LLM training alignment
What is pretraining?
Pretraining (also called Pre-training): The first and most expensive stage of building a large language model, where the network learns from a huge unlabelled text collection by repeatedly predicting missing or next pieces of text. It produces a general-purpose model that knows grammar, facts and patterns, and is later specialised by fine-tuning and alignment.
Pretraining: an example
A team trains a model on trillions of tokens of web pages, books and code for weeks on thousands of GPUs; the result can continue any text but does not yet follow instructions well.
Related terms
More LLM training alignment terms
- Process reward model
- Prompt tuning
- Proximal policy optimisation
- QLoRA
- Rank (LoRA)
- Reasoning model
- Reinforcement learning from AI feedback
- Reinforcement learning from human feedback
Where you learn it
Pretraining is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.