AI engineering glossary · NLP embeddings
What is unigram language model tokeniser?
Unigram language model tokeniser (also called Unigram tokeniser): A subword method that starts with a large candidate vocabulary and repeatedly removes the pieces that least reduce the likelihood of the corpus, until the target size is reached. Each word can be split several ways, and the most probable split is chosen, or one is sampled for regularisation.
Unigram language model tokeniser: an example
"nation" might be split as ["nat", "ion"] or ["na", "tion"]; the tokeniser picks the split with the higher probability.
Related terms
More NLP embeddings terms
- Viterbi algorithm
- Vocabulary
- Vocabulary size
- Word analogy
- Word embedding
- Word sense disambiguation
- Word tokenisation
- Word2Vec
Where you learn it
Unigram language model tokeniser is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.