Skip to content
Back to search results

Part-of-Speech Tagging

Introduced by
Computational and corpus linguists (Penn Treebank; CLAWS tradition)
Year
1993
Type
Sequence labeling of grammatical word categories
Family
Corpus annotation / NLP
Font size

What does it do?

Part-of-speech (POS) tagging is the task of assigning each word (token) in a text its grammatical category — noun, verb, adjective, preposition, and finer distinctions such as past-tense verb or comparative adjective — drawn from a fixed tagset.

Because the same word form can belong to different categories depending on context ("book a flight" versus "read a book"), tagging is fundamentally a disambiguation problem solved with contextual evidence. It is one of the oldest and most foundational tasks in natural language processing and corpus linguistics, supplying the grammatical layer on which concordancing, parsing, register analysis, and information extraction all depend. Modern taggers reach accuracies well above 97% on standard English benchmarks, using statistical sequence models or neural networks trained on annotated corpora.

The intuition behind the method

Reading a sentence, a human effortlessly knows that in "They can fish", "can" is a modal verb and "fish" is its main verb, whereas in "a can of fish" both words are nouns. The cue is context: a word's neighbors and the tags around it constrain what category it can be. A POS tagger captures exactly this intuition statistically — it learns that a word following "the" is usually a noun or adjective, that "to" before a base verb signals an infinitive, and that some words are simply more often one category than another. Tagging is therefore a balance between what a word usually is and what its company says it must be here.

History

Automatic grammatical tagging began with the rule-based TAGGIT system used to tag the Brown Corpus in the 1960s–70s, followed by the probabilistic CLAWS tagger developed at Lancaster (UCREL) to tag the LOB Corpus and later the 100-million-word British National Corpus. The release of the Penn Treebank in 1993 (Marcus, Marcinkiewicz, and Santorini) provided a large, consistently annotated training corpus and a widely adopted tagset, catalyzing the statistical era of taggers — hidden Markov models, Brill's transformation-based learning, maximum-entropy and CRF models surveyed in Jurafsky and Martin. Neural sequence models and contextual embeddings later pushed accuracy still higher, and POS tagging remains a standard component of corpus-annotation and NLP toolchains.

When to use it

Use POS tagging whenever a downstream task needs to know the grammatical category of words: building part-of-speech-annotated corpora, running tag-based concordance or collocation queries ("all instances of run as a noun"), counting grammatical features for register or stylometric analysis, feeding a syntactic parser, or filtering candidate terms in information extraction. It is essential preprocessing for most corpus-linguistic and NLP pipelines on English and many other languages. It is less helpful, or needs careful adaptation, for languages with rich morphology or no clear word boundaries, for very noisy social-media text, and for tasks where fine grammatical distinctions add error without analytic value.

Limitations and considerations

  • Accuracy drops sharply on text unlike the training data — historical, learner, dialectal, or social-media language.
  • Errors concentrate on genuinely ambiguous words and on unknown words, and these errors propagate to downstream tasks.
  • Tagset granularity is a fixed trade-off: finer tags capture more grammar but introduce more disambiguation errors.
  • Morphologically rich and segmentation-free languages need adapted tagsets and tokenizers, reducing the portability of English-centric methods.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Provides a foundational grammatical layer that almost every other corpus and NLP task can build on.
  • Achieves very high accuracy (often above 97% on standard English) with mature, fast, well-understood models.
  • Resolves lexical category ambiguity using context, far beyond what dictionary lookup alone can do.
  • Standard tagsets such as Penn Treebank and CLAWS make annotations comparable and reusable across projects and tools.

Applications

  • Corpus annotation: producing POS-tagged reference corpora such as the BNC (CLAWS) for linguistic research.
  • Tag-based corpus queries: retrieving a word only in a given grammatical role, or searching by grammatical pattern.
  • Feature extraction for register and stylometric studies, including the feature counts in multidimensional register analysis.
  • Preprocessing for syntactic parsing, named-entity recognition, and information extraction pipelines.
  • Lexicography and language teaching: distinguishing senses and uses of a word by its grammatical behavior.

How it works

How it works

1

The text is first segmented into tokens, and a tagset is fixed — the inventory of labels the tagger may assign. The Penn Treebank tagset (about 36 word tags plus punctuation) is the de facto standard for English, distinguishing for example NN (singular noun), NNS (plural noun), VBD (past-tense verb), and JJR (comparative adjective); the British National Corpus uses the richer CLAWS C5/C7 tagsets. The choice of tagset fixes the granularity of the analysis and must match any training data and downstream consumers of the annotation.

2

The tagger models the conditional distribution of tag sequences given the word sequence, P(t_1…t_n | w_1…w_n). Early rule-based and transformation-based taggers encoded this with hand-written and learned rules; statistical taggers represent it as a hidden Markov model (HMM) factoring into emission and transition probabilities, or as a conditional random field (CRF) and maximum-entropy model that can mix arbitrary features such as suffixes, capitalization, and surrounding words. Contemporary neural taggers learn contextual word representations (BiLSTMs or transformers) that encode the whole sentence before predicting tags.

Formula
P(t_1\cdots t_n \mid w_1\cdots w_n)
3

Given the model, the tagger searches for the single most probable tag sequence for the sentence. In the HMM formulation this is the sequence maximizing the product of emission probabilities P(w_i | t_i) and transition probabilities P(t_i | t_{i-1}), found efficiently with the Viterbi dynamic-programming algorithm rather than by enumerating all tag combinations. CRF and neural taggers likewise decode globally (Viterbi over emission and transition scores), ensuring the chosen tags are mutually consistent across the sentence rather than picked independently per word.

Formula
\hat{t}_{1:n} = \arg\max_{t_{1:n}} \prod_{i} P(w_i \mid t_i)\,P(t_i \mid t_{i-1})
4

Tagger quality is measured by accuracy — the proportion of tokens assigned the correct tag against a gold-standard test set — typically reported both overall and on the harder subset of ambiguous or unknown words. Once validated, the tagger is applied to raw corpora to produce annotation that feeds downstream tasks: lemmatization, syntactic parsing, tag-based concordance queries, the feature counts of multidimensional register analysis, and information extraction. Because errors propagate, the reliability of POS tags constrains everything built on top of them.

Common pitfalls

  • Reporting only overall accuracy, which is dominated by unambiguous tokens and hides poor performance on hard cases.
  • Applying an English Penn-Treebank tagger to another language or to noisy text without retraining or adaptation.
  • Trusting tags blindly in downstream analysis without spot-checking, so systematic tagging errors become silent data errors.
  • Mismatching tagsets between training data, tagger, and analysis, producing labels that cannot be compared or interpreted.

Frequently asked questions

Why is POS tagging not just dictionary lookup?

Many word forms belong to more than one category — "book", "run", "that", and "like" can each be several parts of speech. A dictionary lists the possibilities, but only context tells you which one applies in a given sentence. POS tagging is precisely the task of choosing the correct category from those possibilities using the surrounding words and tags, which is why it is modeled as sequence disambiguation rather than a lookup.

What is the Penn Treebank tagset?

The Penn Treebank tagset is a set of about 36 part-of-speech labels (plus punctuation tags) introduced with the Penn Treebank in 1993. It makes fine grammatical distinctions — for instance separating singular nouns (NN) from plural (NNS), and base, past, and gerund verb forms (VB, VBD, VBG). Its wide adoption as training and evaluation data made it the standard tagset for English NLP, though corpora such as the BNC use the alternative CLAWS tagsets.

How accurate are modern POS taggers?

On standard English benchmarks like the Penn Treebank, well-trained statistical and neural taggers exceed 97% token accuracy. However, that figure is inflated by the many words that are never ambiguous; accuracy on genuinely ambiguous tokens and on unknown words is lower, and overall accuracy falls considerably on text that differs from the training data, such as historical, learner, or social-media language.

Related methods

Sources

  1. Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press.

    ISBN: 9780521865715
    Copied.
  2. Jurafsky, D., & Martin, J. H. (2023). Speech and Language Processing (3rd ed. draft). Stanford University.

    Open publication reference
    Copied.
  3. Marcus, M. P., Marcinkiewicz, M. A., & Santorini, B. (1993). Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2), 313–330.

    Open publication reference
    Copied.