AI engineering glossary · NLP embeddings
What is special tokens?
Special tokens (also called Control tokens): Reserved vocabulary entries that carry structure rather than ordinary text, such as markers for padding, unknown words, start or end of a sequence, or a separator between two sentences. They are added by the tokeniser and tell the model how to read the input.
Special tokens: an example
A sentence pair may be encoded as [CLS] first sentence [SEP] second sentence [SEP], where [CLS] and [SEP] are special tokens.
Related terms
More NLP embeddings terms
- Spelling correction
- Static embedding
- Subword regularisation
- Subword tokenisation
- Term frequency
- Text classification
- Text embedding model
- Text normalisation
Where you learn it
Special tokens is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.