AI engineering glossary · LLM training alignment
What is teacher-student learning?
Teacher-student learning: The setup used in distillation: a trained, usually larger teacher model provides targets, and a smaller student model learns to reproduce them. The student may match output distributions, hidden features or generated text.
Teacher-student learning: an example
A large multilingual model labels text and a small model for a phone app learns from those labels.
Related terms
More LLM training alignment terms
- Tensor parallelism
- Test-time compute scaling
- Trust region policy optimisation
- UltraFeedback
- Value function
- Warmup-stable-decay schedule
- Weak-to-strong generalisation
- ZeRO
Where you learn it
Teacher-student learning is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.