AI engineering glossary · NLP embeddings
What is character-level tokenisation?
Character-level tokenisation (also called Character tokenisation): Treating every single character as a token. The vocabulary is tiny and no word is ever unknown, but sequences become very long and each token carries little meaning, so models need more computation to learn word-level patterns.
Character-level tokenisation: an example
"chai" becomes ["c", "h", "a", "i"], four tokens instead of one.
Related terms
More NLP embeddings terms
- Chunking
- Co-occurrence matrix
- Code-mixing
- Collocation
- Conditional random field
- CoNLL-2003
- Constituency parsing
- Contextual embedding
Where you learn it
Character-level tokenisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.