LDA Topic Model
- Introduced by
- Blei, D. M., Ng, A. Y., & Jordan, M. I.
- Year
- 2003
- Type
- Probabilistic generative topic model
- Family
- Deep learning / NLP / CV
What does it do?
Latent Dirichlet Allocation (LDA) is a probabilistic generative model introduced by Blei, Ng, and Jordan in 2003 that discovers hidden thematic structure in large text collections by representing each document as a mixture of latent topics and each topic as a probability distribution over vocabulary words.
The intuition behind the method
Imagine reading a newspaper article about climate change — it might touch on science, politics, and economics simultaneously. LDA formalises this intuition: it assumes every document is a blend of a small number of abstract themes (topics), and every topic is characterised by the words that tend to appear in it. The model works backwards from the observed words to infer both the hidden topics and how much each document reflects each topic, using Bayesian inference with Dirichlet priors to control how spread-out or focused these mixtures are.
History
LDA was introduced by David Blei, Andrew Ng, and Michael Jordan in a landmark 2003 paper in the Journal of Machine Learning Research, building on earlier probabilistic latent semantic analysis (pLSA) by Hofmann (1999). It became the dominant topic modelling technique for over a decade and spawned a rich family of extensions — correlated topic models, dynamic topic models, and author-topic models. In recent years embedding-based approaches such as BERTopic have complemented but not replaced LDA, which remains the standard baseline in text mining and computational social science.
When to use it
LDA is well-suited to exploratory thematic analysis of large unlabelled text corpora — survey responses, news archives, scientific abstracts, social media feeds, or policy documents — where the goal is to discover recurring themes without pre-defined categories. Use it when you have at least several hundred documents and want interpretable, human-readable topic summaries. Prefer LDA over NMF when you want a principled probabilistic framework with uncertainty estimates and tunable sparsity. Do not use LDA when you have short texts (tweets, titles) where each document contains very few words, when you need supervised classification with known labels (use BERT-based classifiers instead), when document order or syntax matters (LDA ignores word order entirely), or when your corpus is smaller than a few hundred documents.
Limitations and considerations
- Bag-of-words assumption ignores word order and sentence structure, losing important syntactic and semantic context.
- The number of topics K must be specified in advance; selecting K poorly yields either incoherent merged topics or overly granular splits.
- Topics are not guaranteed to be semantically coherent and often require careful stop-word tuning and preprocessing.
- Performs poorly on short texts (fewer than ~50 words per document) because there is not enough word co-occurrence signal.
- Inference via Gibbs sampling is stochastic and can produce different topic orderings across runs, complicating replication.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Discovers interpretable latent themes from unlabelled text without requiring any manually labelled training data.
- Probabilistic framework provides per-document topic proportions, enabling nuanced mixed-membership representation.
- Scales to very large corpora using online or distributed variants such as online LDA.
- Tunable Dirichlet priors allow explicit control over topic sparsity and document-topic granularity.
- Topic-word distributions serve as compact, human-readable summaries that facilitate qualitative interpretation.
- Well-established theoretical foundation and broad tool support (Gensim, scikit-learn, MALLET).
Applications
- Thematic analysis of open-ended survey responses and interview transcripts in social sciences.
- Discovering research trends in large bodies of scientific literature or patent filings.
- Content analysis of news archives, political speeches, and social media streams.
- Generating interpretable document features for downstream clustering or classification pipelines.
- Tracking how topics evolve over time using dynamic topic model extensions.
- Automated tagging and recommendation in digital library and content management systems.
How it works
How it works
The corpus is tokenised, lowercased, stripped of stop words, and converted to a bag-of-words representation. Each document becomes a vector of word counts and a shared vocabulary (dictionary) is built. Preprocessing quality — especially stop-word removal and optional stemming or lemmatisation — strongly affects topic coherence.
Two Dirichlet distributions are specified with hyperparameters alpha and beta. Alpha controls document-topic sparsity (low alpha means each document focuses on fewer topics); beta controls topic-word sparsity (low beta means each topic focuses on fewer words). These priors are then used to initialise per-document topic distributions theta and per-topic word distributions phi.
Formula
\theta_d \sim \text{Dir}(\alpha),\quad \phi_k \sim \text{Dir}(\beta)For each word position in a document, LDA samples a topic assignment from the document's topic distribution. This step is the core generative story: a topic is chosen proportional to how dominant it already is in that document.
Formula
z_{d,n} \sim \text{Categorical}(\theta_d)Given the sampled topic for a word position, the actual word is drawn from that topic's word distribution. Topics that are about technology will place high probability on words like 'algorithm' or 'network'; topics about health will favour 'patient' or 'treatment'.
Formula
w_{d,n} \sim \text{Categorical}(\phi_{z_{d,n}})Because the true topic assignments are latent, they must be estimated from observed words. The standard approach is collapsed Gibbs sampling (or variational Bayes), which iteratively re-assigns each word to its most probable topic given current assignments of all other words, repeating until the assignments stabilise and the model converges.
Once inference converges, each topic is summarised by its top-N most probable words, and each document is described by its topic proportions. The researcher labels each topic by inspecting its top words, then uses per-document topic weights as features for downstream analysis such as clustering, classification, or trend detection.
Common pitfalls
- Choosing K arbitrarily without evaluating coherence scores (e.g., C_v) or held-out perplexity across a range of values.
- Skipping stop-word removal or including very high-frequency words that dominate every topic and obscure meaningful themes.
- Applying LDA to very short texts (tweets, titles) where sparse word counts prevent reliable co-occurrence estimation.
- Reporting topic labels without showing the top words and letting readers verify the interpretation independently.
- Treating topic assignments as ground-truth labels and applying them to downstream tasks without assessing topic quality first.
Frequently asked questions
How do I choose the number of topics K?
A principled approach is to fit LDA for a range of K values (e.g., 5 to 50) and evaluate each using a topic coherence score such as C_v computed on a held-out portion of the corpus. The elbow point where coherence stabilises or peaks is a reasonable choice. Human inspection of top words is still essential — coherence metrics guide but do not replace qualitative judgement.
LDA vs. NMF — when should I prefer one over the other?
Both decompose a document-term matrix into topics. LDA is generative and probabilistic, providing uncertainty estimates and explicit sparsity control via Dirichlet priors; it tends to produce more stable topics on larger corpora. NMF is deterministic and faster, often yielding sharper, more focused topics on smaller or sparser matrices. Try both and compare coherence scores.
Can I use LDA on short texts like tweets?
Standard LDA performs poorly on very short texts because each document contains too few words for reliable co-occurrence estimation. Alternatives include aggregating short texts into pseudo-documents before running LDA, using the Biterm Topic Model (BTM) designed for short texts, or switching to embedding-based methods such as BERTopic.
How do I validate that my topics are meaningful?
Compute an intrinsic coherence score (C_v or NPMI) and compare it across K values. Complement this with a word intrusion test — ask domain experts whether a randomly inserted alien word stands out — and inspect top-10 words per topic for semantic consistency. Also examine per-document topic distributions for face validity with known document content.
Does LDA understand word meaning or context?
No. LDA uses a bag-of-words representation and treats each word as an atomic symbol; it has no knowledge of synonymy, polysemy, or word order. Words like 'bank' (financial) and 'bank' (river) are the same token. For context-sensitive modelling, consider word-embedding-enriched topic models or BERTopic, which clusters contextual sentence embeddings.
Related methods
Sources
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet Allocation. Journal of Machine Learning Research, 3, 993–1022.
Open publication referenceCopied.Latent Dirichlet Allocation. Wikipedia.
Open publication referenceCopied.