AI engineering glossary · NLP embeddings
What is sentencePiece?
SentencePiece: A tokeniser library that learns subword units directly from raw text, treating the input as a stream of characters including spaces, so it needs no language-specific pre-splitting. It supports byte-pair encoding and unigram methods, which makes it convenient for languages that do not separate words with spaces.
SentencePiece: an example
A single SentencePiece model can be trained on mixed Hindi, Tamil and English text without writing separate word-splitting rules.
Related terms
More NLP embeddings terms
- Sentiment analysis
- Sequence labelling
- Siamese network
- Skip-gram
- Slot filling
- Smoothing
- spaCy
- Special tokens
Where you learn it
SentencePiece is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.