AI engineering glossary · NLP embeddings
What is vocabulary size?
Vocabulary size: The number of distinct tokens a tokeniser holds. A larger vocabulary shortens sequences but increases the size of the embedding table and gives rare tokens too few training examples; a smaller one lengthens sequences and raises compute cost.
Vocabulary size: an example
Moving from a 30,000-token to a 100,000-token vocabulary with 768-dimensional embeddings raises the embedding table from about 23 million to 77 million numbers.
Related terms
More NLP embeddings terms
- Word analogy
- Word embedding
- Word sense disambiguation
- Word tokenisation
- Word2Vec
- WordPiece
- Zipf's law
- Agglutinative language
Where you learn it
Vocabulary size is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.