AI engineering glossary · NLP embeddings
What is tokenisation?
Tokenisation (also called Tokenization, Tokenizer): The step that splits raw text into tokens and maps each token to an integer identifier from a fixed list. It is the first stage of almost every language pipeline, and the choice of method affects vocabulary size, sequence length, handling of rare words and cost of processing.
Tokenisation: an example
"I love biryani!" split on spaces and punctuation gives ["I", "love", "biryani", "!"], which are then mapped to IDs such as [40, 812, 15007, 3].
Tokenisation in Python
A short example you can run (Python 3.11). The output below is what it really prints.
import re
from collections import Counter
text = "Synaptova teaches AI engineering. AI engineering is learnable!"
tokens = re.findall(r"\w+|[^\w\s]", text.lower())
vocab = {tok: i for i, tok in enumerate(sorted(set(tokens)))}
print(tokens)
print([vocab[t] for t in tokens])
print(Counter(tokens).most_common(2))Output
['synaptova', 'teaches', 'ai', 'engineering', '.', 'ai', 'engineering', 'is', 'learnable', '!']
[6, 7, 2, 3, 1, 2, 3, 4, 5, 0]
[('ai', 2), ('engineering', 2)]Related terms
More NLP embeddings terms
- Tokenizer fertility
- Topic modelling
- Transliteration
- Trigram
- Triplet loss
- Truncation
- Unicode normalisation
- Unigram
Where you learn it
Tokenisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.