TF-IDF — Term Frequency–Inverse Document Frequency
- Introduced by
- Salton & Buckley
- Year
- 1988
- Type
- Text vectorization / term-weighting scheme
What does it do?
TF-IDF, introduced by Salton and Buckley (1988), is a term-weighting scheme that scores each word in a document by how often it appears there and how rare it is across the whole collection.
It turns raw text into weighted document vectors, giving high weight to terms that are frequent in one document but uncommon elsewhere.
The intuition behind the method
Not every word that appears often is informative — words like "the" appear everywhere and tell you little about what a document is about. TF-IDF captures the idea that a term matters most when it is frequent inside a specific document yet rare across the rest of the collection. By multiplying how often a term shows up locally by how distinctive it is globally, the scheme highlights the words that actually characterise each document and downweights the common ones.
History
TF-IDF emerged from the information-retrieval work of Salton and Buckley (1988), who systematically compared term-weighting approaches for automatic text retrieval and established the combination of term frequency with inverse document frequency as an effective document-representation scheme. It became a foundational baseline for text vectorization and remains widely used as a transparent feature representation.
When to use it
Use TF-IDF when you have a text corpus and need to turn documents into numeric features for retrieval, classification, or similarity comparison. It relies on the bag-of-words assumption and requires text preprocessing beforehand. The collection should be reasonably large — with under about 100 documents the IDF weights become unstable and a simple word-frequency analysis is more reliable. With no text data at all, TF-IDF cannot be computed.
Limitations and considerations
- Rests on the bag-of-words assumption and ignores word order and context.
- IDF weights become unstable and meaningless on very small corpora.
- Requires careful text preprocessing; noisy tokens degrade the weights.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Simple, fast, and interpretable — each weight is directly traceable to local frequency and global rarity.
- Turns raw text into numeric document vectors usable by search, classification, and clustering methods.
- Downweights ubiquitous, uninformative words while highlighting terms that distinguish a document.
Applications
- Representing documents as feature vectors for text classification.
- Ranking and retrieving documents in search applications.
- Identifying the most distinctive keywords that characterise a document.
- Building inputs for document clustering and similarity comparison.
How it works
How it works
The text corpus is cleaned and tokenised into terms — text is normalised and split into the word units that will be weighted. Because TF-IDF rests on the bag-of-words assumption, consistent preprocessing across all documents is what makes the resulting counts comparable.
For each document, the term frequency (TF) counts how often each term occurs in that document. A term that appears many times within a single document gets a higher local weight, reflecting how central it is to that document's content.
Across the whole collection, the inverse document frequency (IDF) measures how rare each term is — terms appearing in many documents receive low IDF, while terms confined to few documents receive high IDF. This is what suppresses ubiquitous, uninformative words and rewards distinctive ones.
The TF and IDF components are combined into a single weight per term, and each document becomes a vector of these weighted scores. The resulting document-term vectors are the representation downstream tasks — search, classification, clustering — consume.
Common pitfalls
- Running TF-IDF on a corpus under ~100 documents — IDF weights become meaningless, so fall back to a simple word-frequency analysis instead.
- Skipping or inconsistent preprocessing across documents, which makes the term counts incomparable.
- Trying to apply TF-IDF when there is no text data — categorical coding with content analysis is the appropriate route.
- Expecting TF-IDF to capture meaning or context: it weights isolated terms and cannot model word order or semantics.
Frequently asked questions
What do TF and IDF each measure?
Term frequency (TF) measures how often a term appears within a single document, capturing how central it is locally. Inverse document frequency (IDF) measures how rare the term is across the whole collection, downweighting words that appear in many documents. Multiplying them gives high weight to terms that are frequent in one document but uncommon elsewhere.
How large does my corpus need to be?
TF-IDF needs a reasonably large collection — roughly 100 documents or more. Below that, IDF weights become unstable and lose meaning, and a simpler word-frequency analysis is the more reliable choice.
Does TF-IDF understand meaning or context?
No. TF-IDF rests on the bag-of-words assumption: it weights individual terms by frequency and rarity and ignores word order, syntax, and semantics. For context or semantic similarity, embedding methods such as Word2Vec are more appropriate.
What preprocessing does it require?
The corpus must be cleaned and tokenised consistently before weighting — normalising text and splitting it into comparable term units. Skipping this step leaves noisy tokens that degrade the weights.
Related methods
Sources
Salton, G. & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing & Management, 24(5), 513-523.
Open DOI referenceCopied.