Skip to content
Back to search results

Multi-Head Self-Attention

Introduced by
Vaswani, A. et al.
Year
2017
Type
Attention mechanism (Transformer core)
Font size

What does it do?

Multi-head self-attention, introduced by Vaswani and colleagues in 2017, is the mechanism that lets every position in a sequence compute its relationship to all other positions in parallel.

It is the core of the Transformer architecture and the foundation underneath BERT, GPT, and T5.

The intuition behind the method

Earlier sequence models processed text one step at a time, which made long-range relationships hard to capture and slow to compute. Self-attention instead lets each token look directly at every other token at once and decide how much weight to give each one, so meaning that depends on distant context is captured in a single parallel pass.

History

Self-attention was introduced by Vaswani and colleagues in the 2017 paper 'Attention Is All You Need', which proposed the Transformer architecture and dispensed with recurrence entirely. It quickly became the foundation of large pre-trained language models: Devlin and colleagues built BERT on bidirectional Transformers in 2019, and the same mechanism underlies GPT and T5.

When to use it

Use self-attention when modelling text or other sequential data where relationships between distant elements matter and you have enough data to train a large model — at least about 100 observations, with 500 or more strongly preferred. A GPU is recommended, positional encoding is required, and be aware that compute cost grows quadratically with sequence length. On very small samples it cannot learn reliable representations; simpler machine-learning models are safer there.

Limitations and considerations

  • Compute and memory cost grow quadratically with sequence length, making very long sequences expensive.
  • A GPU is effectively required for practical training.
  • Needs positional encoding because attention alone is order-agnostic.
  • On small datasets (n below about 500) it overfits and fails to learn reliable representations.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Captures long-range dependencies directly, without the bottleneck of step-by-step recurrence.
  • Fully parallel computation across positions, which scales well on modern hardware.
  • Multiple heads can attend to different relationship patterns simultaneously.
  • Serves as the shared backbone for BERT, GPT, and T5, with extensive transfer-learning support.

Applications

  • Text classification and other supervised natural-language tasks.
  • Prediction and representation learning over sequential data.
  • Pre-training and fine-tuning of large language models such as BERT and GPT.
  • Any setting where long-range relationships between sequence elements must be modelled.

How it works

How it works

1

Each input position is mapped into three learned vectors — a query, a key, and a value — through separate weight matrices. The query represents what a token is looking for, the key what each token offers, and the value the content that gets passed along.

Formula
Q = XW_Q,\; K = XW_K,\; V = XW_V
2

Every query is compared against all keys via a scaled dot product, scaled by the square root of the key dimension to keep the values stable, and passed through a softmax to produce attention weights. Each position's output is the weighted sum of all value vectors, so it blends information from across the whole sequence at once.

Formula
\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V
3

The attention is run several times in parallel as separate heads, each with its own projections, so different heads can focus on different kinds of relationships. Their outputs are concatenated and projected back together, and because positions are otherwise interchangeable, positional encoding is added so order information is preserved.

Formula
\text{MultiHead}(X) = \text{Concat}(\text{head}_1,\dots,\text{head}_h)W_O

Common pitfalls

  • Training on too little data — under roughly 500 examples the mechanism overfits instead of learning useful representations.
  • Forgetting positional encoding, which leaves the model blind to word order.
  • Underestimating the quadratic cost of long sequences and running out of memory.
  • Reaching for a Transformer on tiny samples where a simpler model such as random forest or XGBoost would do better.

Frequently asked questions

Why is the attention score scaled by the square root of the key dimension?

Without scaling, dot products grow large as the key dimension increases, pushing the softmax into regions with vanishing gradients. Dividing by the square root of the key dimension keeps the scores in a stable range.

What do the multiple heads add?

Each head learns its own query, key, and value projections, so different heads can attend to different kinds of relationships at once. Their outputs are concatenated and projected back together into a single representation.

Why is positional encoding needed?

Self-attention treats positions as interchangeable, so on its own it has no notion of order. Positional encoding injects information about each token's position so the model can use word order.

How much data does it need?

It is data-hungry: with under about 500 examples it overfits and cannot learn reliable representations, and below roughly 100 training a Transformer is not meaningful. For small samples, simpler models such as random forest or XGBoost are preferable.

Related methods

Sources

  1. Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS.

    Open publication reference
    Copied.
  2. Devlin, J. et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL.

    Open publication reference
    Copied.