AI engineering glossary · NLP embeddings
What is Token ID?
Token ID (also called Input ID): The integer that represents one token, equal to its position in the vocabulary. Models take sequences of token IDs as input and look up a vector for each ID; they never handle the token strings directly.
Token ID: an example
If the vocabulary lists ["<pad>", "the", "cat"], then "cat" has token ID 2.
Related terms
More NLP embeddings terms
- Tokenisation
- Tokenizer fertility
- Topic modelling
- Transliteration
- Trigram
- Triplet loss
- Truncation
- Unicode normalisation
Where you learn it
Token ID is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.