AI engineering glossary · NLP embeddings
What is IDF (inverse document frequency)?
Inverse document frequency (also called IDF): A measure of how rare a term is across a collection of documents: terms found in few documents get high values and terms found in nearly all get low values. A common form is idf = log(N / df), where N is the number of documents and df is how many contain the term.
Inverse document frequency: an example
With N = 1000 documents and "the" in 990 of them, idf is tiny; "mitochondria" in 5 gives log(1000 / 5), a much larger weight.
Related terms
More NLP embeddings terms
- Kneser-Ney smoothing
- Massive Text Embedding Benchmark
- Matryoshka representation learning
- Mean pooling
- Morphology
- Multilingual embedding
- N-gram
- N-gram language model
Where you learn it
Inverse document frequency is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.