AI engineering glossary · NLP embeddings
What is token?
Token (also called Text token): The smallest unit of text a language system treats as one piece after splitting. Depending on the method, a token can be a whole word, part of a word, a single character or a punctuation mark. Models never see raw text; they see sequences of tokens converted to numbers.
Token: an example
The sentence "Unhappiness is rare" might become the tokens ["un", "happiness", "is", "rare"] under a subword tokeniser.
Related terms
More NLP embeddings terms
Where you learn it
Token is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.