AI engineering glossary · NLP embeddings
What is truncation?
Truncation: Cutting a token sequence down to a maximum length so it fits the limit a model can accept. Text beyond the limit is dropped, usually from the end, which can lose important information if the limit is small.
Truncation: an example
A 700-token review truncated to 512 tokens loses its final 188 tokens, which may contain the verdict.
Related terms
More NLP embeddings terms
- Unicode normalisation
- Unigram
- Unigram language model tokeniser
- Viterbi algorithm
- Vocabulary
- Vocabulary size
- Word analogy
- Word embedding
Where you learn it
Truncation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.