AI engineering glossary · LLM training alignment
What is sequence packing?
Sequence packing (also called Example packing): Concatenating several short training examples into one fixed-length sequence, separated by end tokens, so that no compute is wasted on padding. Attention masks often stop examples from seeing each other. It raises throughput noticeably when data has many short samples.
Sequence packing: an example
Instead of 8 padded sequences of length 512 each holding a 100-token sample, pack 5 samples into each 512-token row and use far fewer rows.
Related terms
More LLM training alignment terms
- Sequence-level distillation
- Simple preference optimisation
- Soft labels
- Span corruption
- Superficial alignment hypothesis
- Supervised fine-tuning
- Sycophancy
- Synthetic data
Where you learn it
Sequence packing is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.