N-gram Language Model
- Type
- Statistical language model
What does it do?
An n-gram language model is a statistical model that predicts the probability of the next word by looking only at the previous n−1 words.
Described in detail by Jurafsky and Martin (Speech and Language Processing), it provides foundational infrastructure for text generation, spelling correction, and speech recognition.
The intuition behind the method
Language is sequential: the words you have just seen strongly constrain what comes next. Rather than reasoning over an entire sentence at once, an n-gram model makes a simplifying bet — that only the last handful of words matter for predicting the next one. It counts how often word sequences appear in a training corpus and turns those counts into probabilities, so a phrase like 'thank you very' makes 'much' a likely continuation.
History
The n-gram approach is a long-standing cornerstone of statistical natural-language processing, treated as foundational in Jurafsky and Martin's Speech and Language Processing. A key practical concern has always been smoothing: Chen and Goodman's (1999) empirical study compared smoothing techniques and established methods such as Kneser-Ney as strong choices for handling unseen sequences. N-gram models long served as the standard before neural language models became dominant.
When to use it
Use an n-gram model when you have a reasonably large text corpus (the source suggests at least about 100 documents) and a task framed as predicting or explaining word sequences — autocompletion, spelling correction, or as a baseline language model. It needs enough training text to estimate reliable counts, and smoothing must be applied to handle the sparsity of unseen sequences.
Limitations and considerations
- Captures only short-range context (the previous n−1 words) and cannot model long-distance dependencies.
- Suffers from data sparsity: most word sequences are unseen, so smoothing is mandatory.
- Model size and memory grow quickly as n increases, while higher-order n-grams become increasingly sparse.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Simple, fast, and interpretable — probabilities come directly from observed counts.
- A strong, transparent baseline for text generation, spelling correction, and speech recognition.
- Low computational cost compared with neural language models, with no specialised hardware required.
Applications
- Text generation and autocompletion from a learned word-sequence distribution.
- Spelling and grammar correction by scoring how probable candidate sequences are.
- Speech recognition, where a language model ranks competing transcription hypotheses.
- Lightweight baselines for any task that benefits from estimating sequence probabilities.
How it works
How it works
The training corpus is segmented into tokens and sentences are padded so that beginnings and ends of sequences are modelled explicitly. Every overlapping window of n consecutive tokens becomes one n-gram, which is the unit the model will count and score.
The model counts how often each n-gram and its (n−1)-word prefix occur in the corpus. Dividing the n-gram count by its prefix count gives the conditional probability of the final word given the preceding n−1 words — the core estimate of the model.
Because most possible word sequences never appear in any finite corpus, raw counts assign zero probability to unseen n-grams (the sparsity problem). Smoothing methods such as Laplace add-one or Kneser-Ney redistribute probability mass so that unseen but plausible sequences receive a small non-zero probability. This step is essential — without it the model breaks on real text.
Given a context of n−1 words, the model returns a probability distribution over the next word, from which the most likely continuation can be chosen or text can be generated. The same machinery scores how probable an observed sentence is, which is what powers spelling correction and speech recognition.
Common pitfalls
- Skipping smoothing leaves unseen n-grams at zero probability, which collapses the model on real, unseen text.
- Training on too small a corpus (under roughly 100 documents) yields unreliable counts.
- Choosing n too large makes counts sparse and the model brittle; choosing n too small loses useful context.
- Failing to pad sentence boundaries mishandles the start and end of sequences.
Frequently asked questions
What does the 'n' in n-gram mean?
It is the number of consecutive words considered together. A bigram (n=2) conditions the next word on the single previous word, a trigram (n=3) on the previous two. Larger n captures more context but needs far more training data and becomes sparse.
Why is smoothing necessary?
A finite corpus can never contain every plausible word sequence, so many valid n-grams have a count of zero and would receive zero probability. Smoothing methods such as Laplace add-one or Kneser-Ney reassign a small probability to unseen sequences so the model does not break on new text.
How much training data do I need?
The source suggests at least about 100 documents as a minimum. More text gives more reliable counts; higher-order n-grams in particular need substantially more data to avoid sparsity.
How does an n-gram model differ from a neural language model?
An n-gram model estimates next-word probabilities directly from counts over a fixed window of previous words, making it simple, fast, and interpretable. Neural models learn distributed representations and capture longer context, at higher computational cost. N-grams remain a useful transparent baseline.
Related methods
Sources
Jurafsky, D. & Martin, J.H. (2023). Speech and Language Processing, 3rd ed.
Open publication referenceCopied.Chen, S.F. & Goodman, J. (1999). An Empirical Study of Smoothing Techniques for Language Modeling. Computer Speech & Language, 13(4), 359-394.
Open DOI referenceCopied.