AI engineering glossary · NLP embeddings
What is backoff?
Backoff (also called Katz backoff): A strategy for language models that falls back to a shorter context when the longer one has no counts, for example using a bigram when a trigram was never observed. Discounting keeps the final probabilities summing to one.
Backoff: an example
If the trigram "very old chai" was never seen, the model uses the bigram "old chai" instead.
Related terms
More NLP embeddings terms
Where you learn it
Backoff is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.