Skip to content
Back to search results

Named Entity Recognition (NER)

Type
NLP sequence-labelling task
Font size

What does it do?

Named entity recognition (NER) is a natural-language-processing task that automatically detects and labels entities in text — such as people, organisations, locations, and dates.

Surveyed by Nadeau and Sekine (2007) and later advanced with neural architectures by Lample et al. (2016), it turns free-running text into tagged spans that downstream tools can use.

The intuition behind the method

A document mentions names, places, and dates in ordinary prose, but a machine sees only a stream of words. The goal of NER is to find the spans that refer to real-world entities and attach the right type label to each one. Instead of a person reading and highlighting names by hand, the method scans the text, decides which words form an entity, and tags it as a person, an organisation, a location, a date, and so on.

History

Named entity recognition was consolidated as a task in the survey by Nadeau and Sekine (2007), which reviewed the rule-based and feature-based methods of the field. The work of Lample et al. (2016) at NAACL then introduced neural architectures for NER, moving the field toward sequence-labelling models that learn contextual representations rather than relying on hand-built features.

When to use it

NER fits when you have text data and a language model that supports the language of that text, with preprocessing applied first. A reasonable amount of text is needed; with a very small corpus the model performs poorly and simple rule-based keyword extraction is the safer choice. If there is no text data at all, NER cannot run and you should analyse structured content instead.

Limitations and considerations

  • On a small corpus model performance drops, and simple rule-based keyword extraction becomes the better option.
  • Performance depends on the language model supporting the target language of the text.
  • Requires preprocessing; without clean tokenisation entity boundaries are detected unreliably.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Turns unstructured text into structured, typed entity spans that downstream tasks can consume.
  • Spans multiple approaches: rule-based matching for small problems, neural sequence labelling for large ones.
  • Identifies several entity classes — people, organisations, locations, dates — in a single pass over the text.

Applications

  • Extracting people, organisations, and locations from news and reports for indexing.
  • Identifying named entities in clinical or biomedical text for structured records.
  • Mining mentions of companies, places, and dates from large document collections.
  • Feeding tagged entities into downstream information- and relation-extraction pipelines.

How it works

How it works

1

The raw text is cleaned and segmented into analysable units — sentences are split and words are tokenised — so that entity boundaries can be detected consistently. The chosen language model must support the target language for this to work.

2

Each token is converted into features the model can read. Depending on the approach this means hand-written rules and gazetteers, classic feature vectors, or contextual embeddings from a neural model — each giving the tagger the signal it needs to recognise an entity.

3

The model assigns an entity-type label to each token span. A rule-based approach matches patterns and dictionaries, a machine-learning approach uses a trained sequence labeller, and a neural approach applies an architecture that captures context. The output is a set of text spans tagged with types such as person, organisation, location, and date.

4

The tagged spans are scored against labelled data using precision, recall, and F1, and incorrect or missed entities are inspected before the model is relied on. This guards against trusting a tagger that silently misses a whole entity class.

Common pitfalls

  • Running NER on a small corpus (under ~100 documents) where the model underperforms — fall back to rule-based keyword extraction instead.
  • Applying a model that does not support the corpus language, which degrades recognition.
  • Skipping preprocessing so that noisy or poorly tokenised text breaks entity boundaries.
  • Judging the tagger by accuracy alone when an entire entity class is being missed; report precision, recall, and F1 and inspect missed entities.

Frequently asked questions

Rule-based, machine learning, or neural — which approach should I use?

A rule-based approach matches patterns and dictionaries and works on small problems with no labelled data. A machine-learning sequence labeller learns from labelled examples and adapts to your domain. A neural approach uses contextual architectures and generally performs best on large corpora, at higher computational cost.

What happens if my corpus is very small?

On a small corpus (roughly under 100 documents) the NER model's performance drops. In that case a simpler rule-based keyword extraction approach is preferred over a full entity-recognition model.

Does NER work in languages other than English?

Yes, but the chosen language model must support the target language of the text. A model that does not cover the corpus language is a common cause of poor recognition.

How is NER evaluated?

Tagged spans are compared against labelled data using precision, recall, and F1. Inspecting missed and incorrect entities matters, because a tagger can quietly fail to detect an entire entity class while still looking accurate overall.

Related methods

Sources

  1. Nadeau, D. & Sekine, S. (2007). A survey of named entity recognition. Lingvisticae Investigationes.

    Open publication reference
    Copied.
  2. Lample, G. et al. (2016). Neural Architectures for Named Entity Recognition. NAACL.

    Open DOI reference
    Copied.