Morphological Analysis — Stemming and Lemmatization
- Introduced by
- M.F. Porter (Porter stemmer)
- Year
- 1980
- Type
- Text-normalisation preprocessing task
What does it do?
Morphological analysis splits words into their stems and affixes so that different surface forms of the same word can be treated as one.
It covers two complementary approaches — rule-based stemming, such as the Porter (1980) and Snowball algorithms, and dictionary-aware lemmatization — and is a critical text-normalisation step for agglutinative languages such as Turkish and Arabic.
The intuition behind the method
A reader knows that running, runs, and ran all point back to the verb run, but to a machine these are three unrelated strings. Morphological analysis strips the grammatical endings off a word to recover the underlying form, so the many ways a word can appear collapse into a single token. Stemming chops off suffixes by rule; lemmatization looks the word up to return its true dictionary form. Either way, the goal is to stop counting one concept as several different words.
History
Rule-based stemming was popularised by Porter's (1980) suffix-stripping algorithm, which remains a reference point and later evolved into the Snowball family of stemmers. Probabilistic and model-based morphological analysis followed, with Schmid's (1994) decision-tree part-of-speech tagger illustrating the move toward statistical approaches that inform modern lemmatization.
When to use it
Use morphological analysis as a normalisation step whenever you have tokenised text and morphology rules or a model for its language. It is especially valuable for agglutinative languages such as Turkish and Arabic, where a single root generates many surface forms. It needs the text to be tokenised first and a morphology resource for the target language; without those it cannot run reliably.
Limitations and considerations
- Requires that morphology rules or a model already exist for the target language.
- The text must be tokenised before analysis can begin.
- Rule-based stemming can over-stem (merging unrelated words) or under-stem (missing valid variants).
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Collapses inflected variants of a word into a single form, shrinking the vocabulary for downstream analysis.
- Offers a spectrum from fast rule-based stemming (Porter, Snowball) to dictionary-accurate lemmatization.
- Especially powerful for agglutinative languages such as Turkish and Arabic, where many forms share one root.
Applications
- Normalising text before indexing, search, or information retrieval.
- Reducing vocabulary size ahead of text classification or topic modelling.
- Preprocessing agglutinative-language corpora (Turkish, Arabic) where surface forms multiply.
- Cleaning and standardising text features for downstream text-mining pipelines.
How it works
How it works
The text is first segmented into individual word tokens. Morphological analysis operates on words, so the corpus must already be tokenised before any stems or lemmas can be extracted; this is a stated prerequisite of the method.
Each token is examined against the morphology of the target language to separate the stem from its affixes (prefixes, suffixes, inflections). This step requires that morphological rules or a model exist for the language at hand, which matters most in richly inflected, agglutinative languages.
The word is reduced to a canonical form. A stemming approach (Porter, Snowball) strips suffixes by rule to produce a stem that need not be a real word, while a lemmatization approach returns the actual dictionary lemma, optionally guided by the word's part of speech.
The reduced forms are written back into the corpus so that variant surface forms map to one token. This normalisation shrinks the vocabulary and lets downstream text-mining steps treat inflected variants of a word as the same feature.
Common pitfalls
- Stemming can produce stems that are not real words; do not treat a stem as a dictionary form when a true lemma is needed.
- Applying an English stemmer to text in another language gives meaningless results — the morphology resource must match the corpus language.
- Skipping tokenisation feeds malformed tokens into the analyser and corrupts the output.
- Over-stemming merges distinct words into one stem and under-stemming leaves variants apart; choose between stemming and lemmatization to fit how exact your downstream task must be.
Frequently asked questions
What is the difference between stemming and lemmatization?
Stemming strips suffixes by rule (as in the Porter or Snowball algorithms) to yield a stem that need not be a real word, and it is fast. Lemmatization looks the word up — often using its part of speech — to return the actual dictionary lemma, which is more accurate but heavier.
Why does this matter most for languages like Turkish and Arabic?
These are agglutinative languages where a single root can take many affixes and generate a large number of surface forms. Morphological analysis collapses those forms back to the root, which is essential for text normalisation in such languages.
Do I need to tokenise the text first?
Yes. Morphological analysis operates on individual words, so the corpus must already be tokenised. Tokenisation is a stated prerequisite of the method.
What goes wrong if the morphology resource does not match the language?
The output becomes meaningless. The rules or model must be built for the target language; applying, for example, an English stemmer to non-English text produces incorrect stems.
Related methods
Sources
Porter, M.F. (1980). An Algorithm for Suffix Stripping. Program, 14(3), 130-137.
Open DOI referenceCopied.Schmid, H. (1994). Probabilistic Part-of-Speech Tagging Using Decision Trees. Proceedings of the International Conference on New Methods in Language Processing (NEMLAP).
Open publication referenceCopied.