AI engineering glossary · NLP embeddings
What is unicode normalisation?
Unicode normalisation: Rewriting text so that different code-point sequences that look identical become one canonical form. Standard forms named NFC, NFD, NFKC and NFKD exist. Without it, visually equal strings may not match, which breaks search and tokenisation.
Unicode normalisation: an example
The letter "é" can be stored as one code point or as "e" plus a combining accent; normalising to NFC makes both identical.
Related terms
More NLP embeddings terms
- Unigram
- Unigram language model tokeniser
- Viterbi algorithm
- Vocabulary
- Vocabulary size
- Word analogy
- Word embedding
- Word sense disambiguation
Where you learn it
Unicode normalisation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.