AI engineering glossary · NLP embeddings
What is Byte-level BPE?
Byte-level BPE (also called Byte-level byte-pair encoding): A variant of byte-pair encoding that works on the raw bytes of UTF-8 text instead of characters, so the starting vocabulary is only 256 byte values. Any string in any script can be encoded without an unknown token, at the cost of longer sequences for scripts that need several bytes per character.
Byte-level BPE: an example
A Devanagari letter takes three bytes in UTF-8, so an unusual Hindi word may be split into many byte-level tokens if no merges cover it.
Related terms
More NLP embeddings terms
- Byte-pair encoding
- Case folding
- Character-level tokenisation
- Chunking
- Co-occurrence matrix
- Code-mixing
- Collocation
- Conditional random field
Where you learn it
Byte-level BPE is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.