Skip to content
Back to search results

Transformer (NLP)

Introduced by
Vaswani, A. et al.
Year
2017
Type
Attention-based deep neural network
Font size

What does it do?

The Transformer is an attention-based deep learning model, introduced by Vaswani and colleagues in 2017, that performs text classification, named-entity recognition, and language modelling by letting every token in a sequence attend directly to every other token.

It replaced earlier recurrent designs with a self-attention mechanism that processes whole sequences in parallel.

The intuition behind the method

Earlier sequence models read text word by word, struggling to connect words far apart in a sentence. The Transformer instead lets each word look at every other word at once and decide which ones matter for its meaning. This attention mechanism is what lets the model capture long-range context in language and learn rich, reusable representations of text.

History

The Transformer was introduced by Vaswani and colleagues in the 2017 paper Attention Is All You Need, which showed that an architecture built entirely on attention — without recurrence — could match or exceed previous sequence models. It became the foundation for pre-trained language models such as BERT and GPT, which are now the standard starting point for most natural-language processing tasks.

When to use it

A good fit for text classification, named-entity recognition, and language modelling on large text datasets — at least about 500 documents — where deep contextual understanding of language matters. It does not require normally distributed data, but it does assume a large text corpus, and using a pre-trained model (such as BERT or GPT) is recommended. Below roughly 500 examples the attention mechanism cannot learn reliable patterns, and below about 100 a deep model is pointless; classical machine learning such as Random Forest or XGBoost is the safer choice.

Limitations and considerations

  • Needs large text datasets (about 500 examples or more) to learn reliable patterns.
  • Below roughly 100 examples, deep model training is meaningless and classical ML suffices.
  • Training is computationally expensive and typically relies on pre-trained models.
  • Its internal representations are hard to interpret compared with explicit-coefficient models.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Self-attention captures long-range dependencies across a whole sequence, unlike word-by-word recurrent models.
  • Processes sequences in parallel, making training on large text corpora far more scalable.
  • Pre-trained models (BERT, GPT) can be fine-tuned, transferring knowledge to tasks with limited labelled data.
  • Handles diverse language tasks — classification, NER, and language modelling — within one architecture.
  • No distributional assumptions: it does not require normally distributed data.

Applications

  • Text classification, such as sentiment or topic labelling.
  • Named-entity recognition (NER) over documents.
  • Language modelling and next-token prediction.
  • Fine-tuning pre-trained language models for domain-specific text tasks.

How it works

How it works

1

Raw text is split into tokens and each token is turned into a numeric vector (embedding). Because attention has no built-in notion of order, position information is added to these vectors so the model knows where each token sits in the sequence.

2

The self-attention mechanism compares every token with every other token: queries are matched against keys to produce attention weights, which are then used to combine the value vectors. Stacking many such attention layers lets the model build up a contextual representation in which each token is informed by the whole sequence.

Formula
\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)V
3

The contextual representations feed a task-specific head — a label for text classification, per-token tags for named-entity recognition, or next-token probabilities for language modelling. In practice a model pre-trained on large text corpora (such as BERT or GPT) is fine-tuned on the task at hand.

Common pitfalls

  • Applying a Transformer to a small text dataset where attention cannot learn stable patterns.
  • Training from scratch instead of fine-tuning a pre-trained model such as BERT or GPT.
  • Ignoring the sample-size floor and trusting a deep model where classical ML would do better.
  • Treating attention weights as a definitive explanation of the model's reasoning.

Frequently asked questions

How much text data do I need?

At least about 500 examples for the attention mechanism to learn reliable patterns. Below roughly 100, a deep model is not worth training and classical machine learning such as Random Forest or XGBoost is a better choice.

Should I train a Transformer from scratch?

Usually no. Using a pre-trained model such as BERT or GPT and fine-tuning it on your task is recommended, since it transfers knowledge learned from large corpora and needs far less labelled data.

What tasks does it handle?

Text classification, named-entity recognition, and language modelling, all built on the same self-attention architecture.

Why is the attention mechanism important?

It lets every token in a sequence attend directly to every other token, capturing long-range context that earlier word-by-word recurrent models handled poorly.

Related methods

Sources

  1. Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS.

    Open publication reference
    Copied.