Keyword Extraction — Automatic Key Term Identification
- Type
- NLP text-mining task
What does it do?
Keyword extraction is a natural-language-processing task that automatically identifies the words or phrases that best represent the content of a document.
It turns a body of free text into a compact, ranked list of key terms, drawing on statistical, graph-based methods such as TextRank (Mihalcea & Tarau, 2004), or embedding-based methods such as KeyBERT (Grootendorst, 2020).
The intuition behind the method
A long document carries its meaning in a handful of recurring, distinctive terms, but reading every text by hand to find them does not scale. Keyword extraction does this automatically: it looks at how words are used across the text and surfaces the few that best capture what the document is about, ranking them so the most representative terms rise to the top.
History
Automatic keyword extraction matured with graph-based ranking: Mihalcea and Tarau (2004) introduced TextRank, applying a PageRank-style algorithm over a word co-occurrence graph to surface key terms. The field later moved toward embedding-based methods, exemplified by Grootendorst's (2020) KeyBERT, which scores candidate phrases by their semantic similarity to the document.
When to use it
Keyword extraction fits when you have text data and want to summarise or index its content with representative terms, for either description or exploration. The text should be preprocessed before extraction. Statistical extraction is reliable only on a sizeable corpus; on a small corpus (under roughly 100 documents) a simple frequency analysis is the safer choice, and with no text data at all the task cannot run — qualitative content analysis is the alternative.
Limitations and considerations
- Statistical extraction is unreliable on a small corpus; below roughly 100 documents it is not dependable.
- Requires preprocessed text; skipping cleaning degrades the candidate terms.
- Cannot run at all when there is no text data to work with.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Turns long documents into a compact, ranked list of representative terms automatically.
- Flexible across approaches: statistical frequency, graph-based TextRank, or embedding-based KeyBERT under one pipeline.
- Low barrier to use, supporting both quick description and deeper exploration of a corpus.
Applications
- Indexing and tagging documents with representative key terms for search and retrieval.
- Summarising the content of long texts at a glance.
- Exploring large corpora to see what recurring themes the documents share.
- Supporting downstream tasks such as topic exploration and content organisation.
How it works
How it works
The raw text is cleaned and standardised first — lowercasing, removing noise, and segmenting the document into analysable units. This preprocessing step is required, and a domain-specific dictionary, where one exists, raises the quality of what follows.
From the cleaned text the method assembles a pool of candidate terms — single words and short phrases (n-grams) that could plausibly serve as keywords. This candidate set is what the scoring stage will rank.
Each candidate is scored for how well it represents the document. A statistical approach weights term frequency, a graph-based approach such as TextRank ranks terms by their connections in a co-occurrence graph, and an embedding-based approach such as KeyBERT scores candidates by their semantic similarity to the document.
Candidates are ordered by their scores and the top terms are returned as the document's keywords or keyphrases. The number returned (for example, the top ten) is chosen to give a concise yet representative summary of the content.
Common pitfalls
- Running statistical extraction on too small a corpus (under ~100 documents) gives unreliable keywords — fall back to a simple frequency analysis instead.
- Attempting extraction with no text data at all; use qualitative content analysis in that case.
- Skipping preprocessing, which lets noise and inconsistent tokens distort the candidate pool.
- Ignoring an available domain dictionary, which would otherwise raise extraction quality.
Frequently asked questions
Statistical, TextRank, or KeyBERT — which approach should I use?
A statistical approach weights term frequency and is the simplest. TextRank builds a co-occurrence graph and ranks terms by their connections, capturing structure beyond raw counts. KeyBERT uses contextual embeddings to score candidates by semantic similarity to the document, which tends to surface more meaningful phrases at higher computational cost.
How much text do I need?
Statistical keyword extraction is only reliable on a sizeable corpus. On a small corpus — roughly under 100 documents — a simple frequency analysis is the safer choice. With no text data at all, the task cannot run.
Do I need to preprocess the text first?
Yes. Preprocessing is assumed: the text must be cleaned and standardised before extraction, or noisy tokens will distort the candidate terms. A domain-specific dictionary, where available, further improves quality.
Does it return single words or phrases?
Either. The candidate pool can include single words and short phrases (n-grams), so the output is a ranked list of keywords or keyphrases depending on the configuration.
Related methods
Sources
Mihalcea, R. & Tarau, P. (2004). TextRank: Bringing Order into Texts. EMNLP, 404-411.
Open publication referenceCopied.Grootendorst, M. (2020). KeyBERT. Zenodo.
Open publication referenceCopied.