AI engineering glossary · LLM training alignment
What is soft labels?
Soft labels (also called Soft targets): Target probability distributions over classes or tokens, as opposed to a single correct answer. In distillation the teacher's soft labels reveal how similar wrong answers are, giving the student richer information than one-hot labels.
Soft labels: an example
Teacher gives 'cat' 0.85, 'lynx' 0.10, 'dog' 0.05, so the student learns that a lynx is closer to a cat than a dog is.
Related terms
More LLM training alignment terms
- Span corruption
- Superficial alignment hypothesis
- Supervised fine-tuning
- Sycophancy
- Synthetic data
- Task arithmetic
- Teacher-student learning
- Tensor parallelism
Where you learn it
Soft labels is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.