AI engineering glossary · NLP embeddings
What is tokenizer fertility?
Tokenizer fertility (also called Fertility): The average number of tokens a tokeniser produces per word of a language. High fertility means the language is split into many small pieces, making text longer, slower and costlier to process than languages the tokeniser was designed around.
Tokenizer fertility: an example
If English averages 1.3 tokens per word but Malayalam averages 4.5, Malayalam text costs about three and a half times more tokens for the same word count.
Related terms
More NLP embeddings terms
- Topic modelling
- Transliteration
- Trigram
- Triplet loss
- Truncation
- Unicode normalisation
- Unigram
- Unigram language model tokeniser
Where you learn it
Tokenizer fertility is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.