Attention Mechanism
- Introduced by
- Bahdanau, D.; Luong, M.T.
- Year
- 2015
- Type
- Neural attention layer (encoder-decoder)
What does it do?
The attention mechanism, introduced by Bahdanau, Cho and Bengio in 2015 and refined by Luong, Pham and Manning the same year, lets a sequence decoder dynamically learn which of the encoder's outputs to focus on at each step.
Before the Transformer, it substantially improved machine-translation quality by freeing models from compressing an entire input into a single fixed vector.
The intuition behind the method
A plain encoder-decoder squeezes a whole input sequence into one fixed context vector, which becomes a bottleneck for long sequences. Attention removes that bottleneck: at every output step the decoder looks back over all encoder states and computes a weighted blend, leaning hard on the few positions that matter for the word it is about to produce. It is like a translator glancing back at the most relevant source words rather than trying to remember the entire sentence at once.
History
The attention mechanism was introduced by Bahdanau, Cho and Bengio in 2015 for neural machine translation, jointly learning to align and translate so the model no longer had to compress the whole source sentence into one vector. The same year Luong, Pham and Manning proposed effective dot-product (multiplicative) variants. Attention substantially raised translation quality and became the conceptual seed for the Transformer architecture that followed.
When to use it
Use attention when you have a sequence-to-sequence problem — such as machine translation or other text and continuous-sequence tasks — where prediction or interpretable alignment matters and a sequence-to-sequence backbone is already in place. It assumes a working seq2seq architecture and that a context vector can be computed. Plan for at least a few hundred examples; with roughly 500 or fewer the mechanism struggles to learn meaningful weights, and below about 100 attention-based training is not worthwhile — simpler models such as Random Forest or XGBoost are safer.
Limitations and considerations
- Requires an existing seq2seq backbone and the ability to compute a context vector.
- On small datasets (around 500 examples or fewer) it cannot learn meaningful weights and tends to overfit.
- Below roughly 100 observations attention-based training is essentially meaningless.
- Adds computational cost on top of the underlying encoder-decoder model.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Removes the fixed-vector bottleneck, greatly improving handling of long sequences.
- Attention weights provide an interpretable alignment between input and output.
- Dynamically focuses on the most relevant encoder positions at each decoding step.
- Substantially improved machine-translation quality and paved the way for the Transformer.
Applications
- Neural machine translation between language pairs.
- Sequence-to-sequence text generation and summarisation.
- Prediction over continuous sequences and time-series style inputs.
- Producing interpretable input-output alignments for analysis.
How it works
How it works
At each decoding step the current decoder state is compared against every encoder output to produce a relevance score. Bahdanau-style (additive) attention learns this score with a small feed-forward network, while Luong-style (multiplicative) attention uses a dot-product between query and keys, which is cheaper to compute.
Formula
e_{ij} = \text{score}(s_{i}, h_{j})The raw scores are passed through a softmax so they become a set of attention weights that are positive and sum to one across the input positions. These weights express how much focus each encoder output should receive for the current step, and can be inspected as an interpretable alignment between input and output.
Formula
\alpha_{ij} = \dfrac{\exp(e_{ij})}{\sum_{k} \exp(e_{ik})}The encoder outputs are combined into a single context vector as their weighted sum, using the attention weights. This dynamic context — different at every step — is fed to the decoder so it can generate the next element while attending to the most relevant parts of the input.
Formula
c_{i} = \sum_{j} \alpha_{ij}\, h_{j}Common pitfalls
- Applying attention to a dataset far too small to learn stable weights, inviting overfitting.
- Treating attention weights as a definitive explanation rather than a useful but partial alignment cue.
- Skipping a seq2seq foundation and expecting attention to work on its own.
- Ignoring that very short inputs gain little from attention over a plain encoder-decoder.
Frequently asked questions
What problem does attention solve?
It removes the fixed-vector bottleneck of plain encoder-decoder models. Instead of compressing the whole input into one context vector, the decoder builds a fresh weighted blend of all encoder states at each step, focusing on the positions that matter most.
What is the difference between Bahdanau and Luong attention?
Bahdanau (additive) attention learns the relevance score with a small feed-forward network, while Luong (multiplicative) attention uses a dot-product between query and keys, which is computationally cheaper.
How much data do I need?
Plan for at least a few hundred examples. With around 500 or fewer the mechanism struggles to learn meaningful weights and overfits, and below roughly 100 attention-based training is not worthwhile — simpler models like Random Forest or XGBoost are better.
Can I read attention weights as an explanation?
The weights form an interpretable alignment that shows where the model focused, which is genuinely useful. Treat it as a helpful cue rather than a complete or definitive explanation of the model's reasoning.
Related methods
Sources
Bahdanau, D., Cho, K. & Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR.
Open publication referenceCopied.Luong, M.T., Pham, H. & Manning, C.D. (2015). Effective Approaches to Attention-based Neural Machine Translation. EMNLP, 1412–1421.
Open DOI referenceCopied.