Lexical Diversity — Measuring Vocabulary Richness
- Type
- Text quantification / lexical richness measurement
What does it do?
Lexical diversity analysis quantifies how varied the vocabulary of a text is — how rich an author's word choice is — using measures such as the type-token ratio (TTR), MTLD, vocd-D, and Yule's K. The MTLD and vocd-D measures were validated by McCarthy and Jarvis (2010), building on earlier work by Tweedie and Baayen (1998) on the stability of lexical-richness measures.
The intuition behind the method
If you count how many distinct words a text uses against how many words it contains in total, you get a sense of its vocabulary richness: a text that keeps repeating the same few words is less diverse than one that reaches for many different words. The catch is that this simple ratio shrinks as a text gets longer, so the method uses length-robust measures and equalises text lengths before comparing one text against another.
History
Quantifying vocabulary richness has a long history in stylometry and corpus linguistics. Tweedie and Baayen (1998) examined how stable various lexical-richness measures are as a function of text length, exposing the limitations of length-sensitive ratios. McCarthy and Jarvis (2010) then validated more sophisticated approaches — MTLD, vocd-D, and HD-D — establishing length-robust measures as the standard for diversity assessment.
When to use it
Use lexical diversity when you have tokenised text and want to describe or compare the richness of its vocabulary — across authors, documents, or groups. It suits descriptive and comparative goals and needs only a modest amount of text (around ten tokens at minimum), but for fair comparison the texts being compared must be put on equal length footing and matched for language and genre.
Limitations and considerations
- Text length affects the TTR calculation; raw TTR is unreliable when comparing texts of different sizes.
- Comparative analysis requires equalised text lengths (standard snippets or rarefaction) to be valid.
- Language and genre shift diversity scores, so uncontrolled comparisons can be misleading.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Turns a text's vocabulary richness into a single comparable number using well-validated measures (TTR, MTLD, vocd-D, Yule's K).
- MTLD and vocd-D are designed to be more robust to text length than the raw type-token ratio.
- Introductory to apply and useful for both descriptive profiling and comparison across authors or groups.
Applications
- Profiling an author's vocabulary richness in stylometric and authorship studies.
- Comparing lexical diversity across writers, documents, or genres in the humanities.
- Assessing vocabulary development in language-learning and educational research.
- Characterising word-usage concentration in social-science and business text corpora.
How it works
How it works
The text is split into tokens (words). Counting distinct word types against the total number of tokens is the foundation every diversity measure builds on, so consistent tokenisation across the texts you intend to compare is the first requirement.
One or more lexical-diversity measures are computed. The type-token ratio (TTR) divides the number of distinct word types by the total tokens; MTLD and vocd-D estimate diversity in ways less sensitive to text length; and Yule's K characterises the concentration of repeated words. Together they describe the richness of the vocabulary from different angles.
Because TTR shrinks as text length grows, raw scores are not directly comparable across texts of different sizes. MTLD and vocd-D are more length-independent by design, but for any comparison the texts should be put on equal footing — for example by equalising lengths with standard-sized snippets or rarefaction — so that differences reflect vocabulary, not document length.
Normalised scores are compared across texts, authors, or groups. Because language and genre also shift diversity, those factors are held constant or controlled for so the comparison isolates the vocabulary-richness signal rather than confounds.
Common pitfalls
- Comparing raw TTR across texts of different lengths — longer texts inevitably show lower TTR, so the comparison reflects length, not vocabulary; use MTLD or vocd-D or equalise lengths first.
- Failing to equalise text lengths in a comparison (no standard snippet or rarefaction) makes the diversity difference uninterpretable.
- Ignoring language and genre differences when comparing texts confounds the vocabulary-richness signal with stylistic factors.
Frequently asked questions
Why not just use the type-token ratio (TTR)?
TTR is intuitive but shrinks as a text grows, because every text eventually reuses common words. That makes raw TTR unreliable for comparing texts of different lengths. MTLD and vocd-D are designed to be far less sensitive to text length, which is why they are preferred for comparison.
How much text do I need?
The method runs on a modest amount of text — around ten tokens at minimum — but more text yields more stable estimates. For comparison, what matters most is that the texts are placed on equal length footing rather than that any single text is very long.
Can I compare the diversity of two different texts directly?
Only after putting them on equal footing. Equalise their lengths with standard-sized snippets or rarefaction, and control for language and genre, because all three factors shift diversity scores independently of vocabulary richness.
Which measure should I report?
MTLD and vocd-D are the validated, length-robust choices and are usually preferred. Yule's K describes how concentrated repeated words are, and TTR remains useful as a simple descriptive figure when texts are length-matched. Reporting more than one measure gives a fuller picture.
Related methods
Sources
McCarthy, P. M. & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods, 42(2), 381-392.
Open DOI referenceCopied.Tweedie, F. J. & Baayen, R. H. (1998). How Variable May a Constant Be? Measures of Lexical Richness in Perspective. Computers and the Humanities, 32(5), 323-352.
Open DOI referenceCopied.