AI engineering glossary · NLP embeddings
What is BPE (byte-pair encoding)?
Byte-pair encoding (also called BPE): A subword method that starts with single characters and repeatedly merges the most frequent adjacent pair of symbols in the training corpus into a new symbol, until a chosen vocabulary size is reached. At use time, the learned merges are applied in order to split new text.
Byte-pair encoding: an example
If "l" and "o" appear together most often, the first merge creates "lo"; later merges may build "low" and "lower".
Related terms
More NLP embeddings terms
- Case folding
- Character-level tokenisation
- Chunking
- Co-occurrence matrix
- Code-mixing
- Collocation
- Conditional random field
- CoNLL-2003
Where you learn it
Byte-pair encoding is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.