N-gram Analysis
- Introduced by
- Corpus linguists (Douglas Biber; lexical bundles tradition)
- Year
- 1999
- Type
- Frequency analysis of contiguous word sequences
- Family
- Corpus linguistics
What does it do?
N-gram analysis is a corpus-linguistic technique that extracts and ranks every contiguous sequence of n words (or characters) in a corpus, exposing the recurrent multi-word units — two-word bigrams, three-word trigrams, and longer 'lexical bundles' — that make up a register or text type.
By counting how often each sequence recurs, it reveals the prefabricated, formulaic backbone of language that single-word frequency lists cannot capture.
The intuition behind the method
Language is not built fresh word by word; speakers and writers reuse ready-made chunks like 'on the other hand', 'I don't know', and 'as a result of'. If you slide a window of fixed width across a corpus and tally every sequence it frames, the chunks that recur far more than chance would predict float to the top. Those high-frequency bundles are the structural scaffolding of a genre — academic prose leans on 'it is important to', conversation on 'do you want to' — and counting them turns an intuition about formulaic language into a ranked, comparable list.
History
Character and word n-grams have roots in Claude Shannon's information theory and in early statistical language modeling. In corpus linguistics, the contiguous-sequence approach was popularized as 'lexical bundles' by Douglas Biber and colleagues in the 1999 Longman Grammar of Spoken and Written English, who showed that registers differ systematically in their recurrent multi-word strings. The Routledge Handbook of Corpus Linguistics consolidated n-gram, cluster, and bundle methods, and tools such as AntConc made cluster/n-gram extraction routine for descriptive corpus work.
When to use it
Use n-gram analysis when you want to characterize the recurrent, formulaic multi-word patterning of a register, genre, or author, or to compare such patterning across corpora — for example, contrasting the lexical bundles of student and expert academic writing. It underpins lexical-bundle research, phraseology, language testing, and statistical language modeling. It is less suited to capturing discontinuous or syntactically flexible patterns (where collocation or collostructional methods are better), and very long n yields sparse counts that need large corpora to be meaningful.
Limitations and considerations
- Only contiguous sequences are captured, so discontinuous patterns ('take ... into account') are split and missed.
- Counts are dominated by frequent function-word sequences unless frequency and dispersion filters are applied.
- Longer n-grams become sparse quickly, demanding very large corpora for stable estimates.
- Frequency alone does not establish that a sequence is psychologically a 'unit'; functional interpretation remains a separate, partly subjective step.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Captures the formulaic, prefabricated multi-word units that single-word frequency lists miss entirely.
- Fully automatic and language-agnostic, requiring no syntactic annotation or theoretical assumptions.
- Produces ranked, normalized counts that support rigorous quantitative comparison across registers and corpora.
- Scales effortlessly to billions of words and underlies practical applications from language modeling to plagiarism detection.
Applications
- Register analysis: comparing the lexical bundles characteristic of conversation, academic prose, and news.
- English for Academic Purposes: identifying the formulaic phrases learners must master for fluent writing.
- Authorship attribution and forensic linguistics: using n-gram profiles as stylistic fingerprints.
- Statistical language modeling and predictive text, where n-gram probabilities estimate the next word.
- Phraseology and second-language learning research on formulaic sequences.
How it works
How it works
The corpus is tokenized into an ordered stream of words (or, in character n-gram work, characters), with decisions made about case-folding, punctuation, and whether sentence or text boundaries block sequences from spanning across them. These preprocessing choices determine which sequences are even eligible to be counted.
A window of width n slides one token at a time across the stream, emitting the contiguous sequence w_i through w_{i+n-1} at each position. Choosing n fixes the granularity: n=2 yields bigrams, n=3 trigrams, and four- to six-word windows yield the lexical bundles studied in register analysis.
Formula
w_{i}\,w_{i+1}\cdots w_{i+n-1}Each distinct sequence g is counted by the number of positions at which it occurs, f(g), and the sequences are ranked by frequency. Normalized frequencies (per million words) make counts comparable across corpora of different sizes, and dispersion measures check that a bundle is spread across many texts rather than concentrated in one.
Formula
f(g) = \#\{i : g = w_i\cdots w_{i+n-1}\}Raw n-gram lists are dominated by trivial sequences, so analysts apply frequency and dispersion thresholds (e.g., a bundle must occur at least 10 times across at least 5 texts) and then interpret the surviving units functionally — classifying lexical bundles as stance, discourse-organizing, or referential, and comparing their profiles across registers.
Common pitfalls
- Reporting raw frequency without normalizing per million words, making cross-corpus comparisons invalid.
- Treating a bundle that occurs many times in a single text as a genre feature, when dispersion across texts is what matters.
- Letting punctuation and sentence boundaries leak into sequences, inflating meaningless cross-boundary n-grams.
- Choosing n without regard to corpus size, so that four- and five-grams are too sparse to support claims.
Frequently asked questions
What is the difference between n-gram analysis and collocation analysis?
N-gram analysis counts contiguous sequences of fixed length and ranks them by raw or normalized frequency, capturing formulaic chunks regardless of statistical attraction. Collocation analysis measures the statistical association between a node word and words in its window (using measures such as mutual information or log-likelihood), can span gaps, and asks whether co-occurrence exceeds chance. N-grams answer 'what recurs', collocation answers 'what is attracted'.
What are lexical bundles?
Lexical bundles are the most frequent recurrent contiguous word sequences (typically three to six words) in a register, identified by frequency and dispersion thresholds. Examples include 'on the other hand' and 'as a result of'. Biber's work classifies them functionally into stance bundles, discourse organizers, and referential bundles, and shows that registers have distinctive bundle profiles.
How do I choose the value of n?
The choice depends on the question and the corpus size. Bigrams and trigrams capture tight collocational chunks and remain frequent enough for small corpora; four- to six-grams capture the lexical bundles studied in register research but require large corpora to occur often enough. Larger n always yields sparser counts, so n is a trade-off between pattern length and statistical reliability.
Related methods
Sources
Biber, D., Johansson, S., Leech, G., Conrad, S., & Finegan, E. (1999). Longman Grammar of Spoken and Written English. Longman.
ISBN: 9780582237254Copied.O'Keeffe, A., & McCarthy, M. (Eds.). (2010). The Routledge Handbook of Corpus Linguistics. Routledge.
ISBN: 9780415464895Copied.Anthony, L. (2004). AntConc: A learner and classroom friendly, multi-platform corpus analysis toolkit. In Proceedings of IWLeL 2004: An Interactive Workshop on Language e-Learning (pp. 7–13). Waseda University.
Open publication referenceCopied.