AI engineering glossary · LLM training alignment
What is instructGPT?
InstructGPT: A 2022 OpenAI research model and paper that fine-tuned GPT-3 with supervised demonstrations and then reinforcement learning from human feedback, showing that a smaller aligned model could be preferred by people over a much larger unaligned one. It established the standard three-stage recipe of SFT, reward model and RL.
InstructGPT: an example
The recipe: collect human-written demonstrations, train a reward model on ranked outputs, then optimise the model against that reward model with PPO.
Related terms
More LLM training alignment terms
- Instruction backtranslation
- Instruction tuning
- Kahneman-Tversky optimisation
- KL penalty
- Knowledge distillation
- LIMA
- Long-context fine-tuning
- LoRA
Where you learn it
InstructGPT is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.