AI engineering glossary · LLM training alignment
What is knowledge distillation?
Knowledge distillation (also called Model distillation, Distillation): Training a smaller student model to imitate a larger teacher model, using the teacher's output probabilities or generated text as targets. The student keeps much of the teacher's ability at lower serving cost and latency.
Knowledge distillation: an example
A 70-billion-parameter teacher's answers train a 7-billion-parameter student that runs on a single GPU.
Related terms
More LLM training alignment terms
- LIMA
- Long-context fine-tuning
- LoRA
- Loss masking
- Loss spike
- Masked language modelling
- Maximal update parametrisation
- Megatron-LM
Where you learn it
Knowledge distillation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.