AI engineering glossary · LLM training alignment
What is span corruption?
Span corruption: A denoising pretraining objective, used by T5, in which contiguous stretches of tokens are replaced by single sentinel placeholders and the model must generate the missing spans in order. It trains an encoder-decoder model to reconstruct text and is a variant of masked language modelling.
Span corruption: an example
The input 'Monsoon <X> Kerala in <Y>' has the target '<X> reaches <Y> June', so the decoder learns to output the removed spans.
Related terms
More LLM training alignment terms
- Superficial alignment hypothesis
- Supervised fine-tuning
- Sycophancy
- Synthetic data
- Task arithmetic
- Teacher-student learning
- Tensor parallelism
- Test-time compute scaling
Where you learn it
Span corruption is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.