Text Frequency Analysis — Word and N-gram Counts
- Introduced by
- George K. Zipf (frequency-distribution foundation)
- Year
- 1949
- Type
- Descriptive text-mining analysis
What does it do?
Text frequency analysis is a descriptive text-mining method that counts how often words, n-grams, and phrases occur in a corpus to reveal content patterns and dominant themes.
It rests on the frequency-distribution insight formalised by George K. Zipf (1949), that a few terms occur very often while most are rare, and it is one of the most basic and widely used entry points into quantitative text analysis.
The intuition behind the method
When you read a body of text, the words that come up again and again tell you what it is about. Text frequency analysis makes that intuition explicit and reproducible: instead of skimming, you tally every token, every two- or three-word sequence, and every recurring phrase, then rank them by how often they appear. The result is a frequency table that surfaces the dominant vocabulary and themes of a corpus at a glance, without any subjective reading.
History
Text frequency analysis builds on Zipf's (1949) observation, in Human Behavior and the Principle of Least Effort, that word frequencies follow a characteristic distribution where a few words occur very often and most are rare. Manning and Schütze (1999), in Foundations of Statistical Natural Language Processing, established frequency counting as a foundational tool of statistical NLP, on which richer methods such as TF-IDF and topic modelling are built.
When to use it
Use text frequency analysis when you have text data and want a descriptive or exploratory first look at its dominant vocabulary, n-grams, or phrases. It suits cross-sectional and longitudinal corpora and needs only a modest amount of text (around 10 documents or more). Proper preprocessing — tokenisation, stop-word removal, and lemmatisation — should be done first, and you should keep in mind that corpus size and language diversity affect the reliability of the frequencies.
Limitations and considerations
- Corpus size and language diversity affect the reliability of the frequencies.
- Raw counts capture surface vocabulary but not context, meaning, or sentiment.
- Results depend heavily on preprocessing choices such as stop-word lists and lemmatisation.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Simple, transparent, and easy to interpret — it is the most basic entry point into quantitative text analysis.
- Reveals content patterns and dominant themes directly from raw counts, with no labelled training data required.
- Works on words, n-grams, and phrases alike, so it adapts to different units of meaning.
Applications
- Identifying the dominant vocabulary and themes of a document collection.
- Exploring n-gram and phrase patterns in survey responses or interview transcripts.
- Profiling content in social, health, education, engineering, and business corpora.
- Serving as a descriptive first step before more advanced text-mining methods.
How it works
How it works
The raw text is prepared before anything is counted: it is tokenised into words, stop-words (high-frequency function words such as 'the' or 'and') are removed, and remaining tokens are lemmatised to a common base form. Consistent preprocessing is what makes the eventual counts comparable and meaningful rather than dominated by grammatical filler.
Each unit of interest is tallied across the corpus. Single tokens are counted for word frequency, adjacent token sequences are counted for n-gram frequency (bigrams, trigrams), and recurring multi-word expressions are counted as phrases. The output of this stage is a raw count for every distinct term.
The counted terms are ordered from most to least frequent, typically reported as a most-common list. This ranked view is where the characteristic distribution appears — a small head of very frequent terms and a long tail of rare ones, consistent with Zipf's law.
The ranked frequencies are read for content patterns and dominant themes. The size of the corpus and its language diversity affect how reliable the counts are, so interpretation accounts for corpus scale and the long tail of rare words rather than over-reading any single count.
Common pitfalls
- Skipping preprocessing lets high-frequency function words and inflected variants dominate the table and obscure the real content.
- Reading frequencies from a corpus that is too small or too narrow gives unstable counts that do not generalise.
- Over-interpreting the long tail — rare terms (the Zipfian tail) carry little frequency signal and are easily mistaken for noise or for meaning.
- Comparing raw counts across corpora of different sizes without normalising leads to misleading conclusions.
Frequently asked questions
What does text frequency analysis actually count?
It counts how often individual words, adjacent word sequences (n-grams such as bigrams and trigrams), and recurring phrases appear across a corpus, then ranks them. The output is a frequency table that surfaces the dominant terms and themes.
Why do I need to preprocess the text first?
Without tokenisation, stop-word removal, and lemmatisation, high-frequency function words and inflected variants of the same word dominate the counts and hide the real content. Consistent preprocessing keeps the frequencies comparable and meaningful.
What is Zipf's law and why does it matter here?
Zipf's law describes the typical pattern that a small number of words occur very frequently while most words are rare, forming a long tail. It explains the shape of the ranked frequency list and warns against over-reading the rare-word tail.
How much text do I need?
A modest corpus of roughly ten documents or more is workable, but corpus size and language diversity affect how reliable the frequencies are — larger and more representative corpora give more stable counts.
Related methods
Sources
Zipf, G. K. (1949). Human Behavior and the Principle of Least Effort. Addison-Wesley.
Open publication referenceCopied.Manning, C. D. & Schütze, H. (1999). Foundations of Statistical Natural Language Processing. MIT Press.
ISBN: 9780262133609Copied.