Named Entity Recognition (NER)
- Type
- NLP sequence-labelling task
What does it do?
Named entity recognition (NER) is a natural-language-processing task that automatically detects and labels entities in text — such as people, organisations, locations, and dates.
Surveyed by Nadeau and Sekine (2007) and later advanced with neural architectures by Lample et al. (2016), it turns free-running text into tagged spans that downstream tools can use.
The intuition behind the method
A document mentions names, places, and dates in ordinary prose, but a machine sees only a stream of words. The goal of NER is to find the spans that refer to real-world entities and attach the right type label to each one. Instead of a person reading and highlighting names by hand, the method scans the text, decides which words form an entity, and tags it as a person, an organisation, a location, a date, and so on.
History
Named entity recognition was consolidated as a task in the survey by Nadeau and Sekine (2007), which reviewed the rule-based and feature-based methods of the field. The work of Lample et al. (2016) at NAACL then introduced neural architectures for NER, moving the field toward sequence-labelling models that learn contextual representations rather than relying on hand-built features.
When to use it
NER fits when you have text data and a language model that supports the language of that text, with preprocessing applied first. A reasonable amount of text is needed; with a very small corpus the model performs poorly and simple rule-based keyword extraction is the safer choice. If there is no text data at all, NER cannot run and you should analyse structured content instead.
Limitations and considerations
- On a small corpus model performance drops, and simple rule-based keyword extraction becomes the better option.
- Performance depends on the language model supporting the target language of the text.
- Requires preprocessing; without clean tokenisation entity boundaries are detected unreliably.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Turns unstructured text into structured, typed entity spans that downstream tasks can consume.
- Spans multiple approaches: rule-based matching for small problems, neural sequence labelling for large ones.
- Identifies several entity classes — people, organisations, locations, dates — in a single pass over the text.
Applications
- Extracting people, organisations, and locations from news and reports for indexing.
- Identifying named entities in clinical or biomedical text for structured records.
- Mining mentions of companies, places, and dates from large document collections.
- Feeding tagged entities into downstream information- and relation-extraction pipelines.
How it works
How it works
The raw text is cleaned and segmented into analysable units — sentences are split and words are tokenised — so that entity boundaries can be detected consistently. The chosen language model must support the target language for this to work.
Each token is converted into features the model can read. Depending on the approach this means hand-written rules and gazetteers, classic feature vectors, or contextual embeddings from a neural model — each giving the tagger the signal it needs to recognise an entity.
The model assigns an entity-type label to each token span. A rule-based approach matches patterns and dictionaries, a machine-learning approach uses a trained sequence labeller, and a neural approach applies an architecture that captures context. The output is a set of text spans tagged with types such as person, organisation, location, and date.
The tagged spans are scored against labelled data using precision, recall, and F1, and incorrect or missed entities are inspected before the model is relied on. This guards against trusting a tagger that silently misses a whole entity class.
Common pitfalls
- Running NER on a small corpus (under ~100 documents) where the model underperforms — fall back to rule-based keyword extraction instead.
- Applying a model that does not support the corpus language, which degrades recognition.
- Skipping preprocessing so that noisy or poorly tokenised text breaks entity boundaries.
- Judging the tagger by accuracy alone when an entire entity class is being missed; report precision, recall, and F1 and inspect missed entities.
Frequently asked questions
Rule-based, machine learning, or neural — which approach should I use?
A rule-based approach matches patterns and dictionaries and works on small problems with no labelled data. A machine-learning sequence labeller learns from labelled examples and adapts to your domain. A neural approach uses contextual architectures and generally performs best on large corpora, at higher computational cost.
What happens if my corpus is very small?
On a small corpus (roughly under 100 documents) the NER model's performance drops. In that case a simpler rule-based keyword extraction approach is preferred over a full entity-recognition model.
Does NER work in languages other than English?
Yes, but the chosen language model must support the target language of the text. A model that does not cover the corpus language is a common cause of poor recognition.
How is NER evaluated?
Tagged spans are compared against labelled data using precision, recall, and F1. Inspecting missed and incorrect entities matters, because a tagger can quietly fail to detect an entire entity class while still looking accurate overall.
Related methods
Sources
Nadeau, D. & Sekine, S. (2007). A survey of named entity recognition. Lingvisticae Investigationes.
Open publication referenceCopied.Lample, G. et al. (2016). Neural Architectures for Named Entity Recognition. NAACL.
Open DOI referenceCopied.