AI engineering glossary · NLP embeddings
What is word tokenisation?
Word tokenisation (also called Word-level tokenisation): Splitting text into whole words, usually using spaces and punctuation rules. It is simple and easy to read, but the vocabulary grows very large, and any word not seen during training becomes an unknown token, which loses information.
Word tokenisation: an example
"Don't stop" may be split into ["Do", "n't", "stop"] or ["Don't", "stop"] depending on the tokeniser's rules.
Related terms
More NLP embeddings terms
Where you learn it
Word tokenisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.