AI engineering glossary · LLM training alignment
What is sequence-level distillation?
Sequence-level distillation (also called Sequence-level knowledge distillation): Distillation in which the student is trained on complete outputs generated by the teacher, for instance its decoded translations or answers, rather than on per-token probabilities. It is the common approach when only text outputs, not logits, are available.
Sequence-level distillation: an example
Collect 200,000 answers from a large model via its API and fine-tune a smaller open model on those texts.
Related terms
More LLM training alignment terms
- Simple preference optimisation
- Soft labels
- Span corruption
- Superficial alignment hypothesis
- Supervised fine-tuning
- Sycophancy
- Synthetic data
- Task arithmetic
Where you learn it
Sequence-level distillation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.