AI engineering glossary · NLP embeddings
What is subword tokenisation?
Subword tokenisation (also called Subword units): A middle path between word and character splitting: frequent words stay whole while rare words are broken into smaller reusable pieces. This keeps the vocabulary manageable, avoids unknown words, and lets the model relate words that share pieces such as prefixes and suffixes.
Subword tokenisation: an example
"tokenisation" could be split as ["token", "isation"], and "tokenising" as ["token", "ising"], sharing the first piece.
Related terms
More NLP embeddings terms
- Term frequency
- Text classification
- Text embedding model
- Text normalisation
- TF-IDF
- Token
- Token ID
- Tokenisation
Where you learn it
Subword tokenisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.