AI engineering glossary · NLP embeddings
What is TFIDF (TF-IDF)?
TF-IDF (also called Term frequency-inverse document frequency, TFIDF): A weighting scheme that multiplies how often a term appears in a document by how rare it is across the collection: tf-idf = tf x idf. It highlights words that characterise a document and suppresses words common everywhere, and it is a classic feature for search and text classification.
TF-IDF: an example
A word with tf 0.03 and idf 2.3 scores 0.03 x 2.3 = 0.069, while "the" with idf near 0 scores almost nothing.
TF-IDF in Python
A short example you can run (Python 3.11, scikit-learn). The output below is what it really prints.
from sklearn.feature_extraction.text import TfidfVectorizer
docs = ["gst invoice for course", "course refund policy", "gst rate and gst number"]
vec = TfidfVectorizer()
X = vec.fit_transform(docs)
row = X[2].toarray()[0]
for word, score in sorted(zip(vec.get_feature_names_out(), row), key=lambda x: -x[1])[:3]:
print(f"{word:8s} {score:.3f}")Output
gst 0.660 and 0.434 number 0.434
Related terms
More NLP embeddings terms
Where you learn it
TF-IDF is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.