Skip to content
Back to search results

Sequence-to-Sequence Model

Introduced by
Sutskever, I.; Cho, K.
Year
2014
Type
Encoder-decoder neural network (deep learning)
Font size

What does it do?

The sequence-to-sequence (Seq2Seq) model, introduced by Sutskever, Vinyals and Le and by Cho and colleagues in 2014, is an encoder-decoder neural network that maps a variable-length input sequence to a variable-length output sequence.

It is the foundation of machine translation, text summarization, dialogue systems and code generation.

The intuition behind the method

Many language tasks turn one sequence into another whose length differs and is not known in advance — a sentence in English becomes a sentence of different length in French. A single classifier cannot do this. Seq2Seq splits the work in two: one network reads the whole input and compresses its meaning into a context representation, and a second network unrolls that representation into the output one token at a time, each new token conditioned on what it has already produced.

History

The encoder-decoder framework emerged in 2014 from two near-simultaneous papers: Sutskever, Vinyals and Le's 'Sequence to Sequence Learning with Neural Networks' and Cho and colleagues' RNN encoder-decoder for statistical machine translation. The original models compressed the source into a single fixed context vector; the subsequent addition of attention removed that bottleneck and dramatically improved translation quality, paving the way for the Transformer architecture that now dominates sequence modelling.

When to use it

Use Seq2Seq when the task is to transform one variable-length sequence into another — machine translation, summarization, dialogue or code generation — and you have paired source-target examples, ideally at least about 500 of them. It suits text and continuous sequences in time-series or cross-sectional form and does not require normally distributed data. A GPU is recommended. With far fewer than 500 pairs the encoder-decoder structure cannot be learned well, and simpler models such as Random Forest or XGBoost are safer.

Limitations and considerations

  • Needs a sizeable corpus of paired source-target examples (about 500 or more) to train well.
  • A GPU is recommended; training is computationally demanding.
  • Compressing a long input into a single fixed context can lose information without attention.
  • On small datasets the encoder-decoder structure cannot be learned and the model overfits.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Handles variable-length inputs and outputs, unlike fixed-output classifiers.
  • A single, general framework for translation, summarization, dialogue and code generation.
  • Trained end to end from paired source-target examples without hand-crafted alignment.
  • Performance improves substantially when an attention mechanism is added.
  • Assumption-light: no normality requirement; works on text and continuous sequence data.

Applications

  • Machine translation between language pairs.
  • Abstractive text summarization.
  • Dialogue and conversational response generation.
  • Code generation from natural-language descriptions.

How it works

How it works

1

An encoder network reads the input sequence token by token and compresses the entire source into a context representation that summarizes its meaning. Because the encoder processes the whole input before any output is produced, the model can handle inputs of any length.

Formula
c = \text{Encoder}(x_1, \dots, x_T)
2

A decoder network generates the output sequence one token at a time. At each step it conditions on the encoder's context together with the tokens it has already emitted, predicting a probability distribution over the next token until a stop symbol ends the sequence. This is what lets the output length differ from the input.

Formula
p(y_t \mid y_{<t}, c)
3

Encoder and decoder are trained jointly to maximize the likelihood of the correct target sequence given the source — supervision comes from paired source-target examples. Adding an attention mechanism, which lets the decoder look back at all encoder states rather than a single fixed context, improves performance, and a GPU is recommended for training.

Formula
\max_{\theta} \sum \log p(y \mid x; \theta)

Common pitfalls

  • Training on too few source-target pairs, where the model overfits and fails to learn the mapping.
  • Omitting an attention mechanism on long sequences and losing distant context.
  • Attempting to train without GPU resources and abandoning before convergence.
  • Using Seq2Seq for a fixed-output task that a simpler classifier or regressor would handle better.

Frequently asked questions

How much data does Seq2Seq need?

It needs paired source-target examples — ideally at least about 500. Below roughly 200 pairs the model overfits and cannot learn the sequence transformation; with very little data a simpler model such as Random Forest or XGBoost is safer.

Do I need a GPU?

A GPU is recommended. Training an encoder-decoder over many sequence pairs is computationally demanding, and CPU-only training is often impractically slow.

What does adding attention do?

Attention lets the decoder look back at all encoder states at each step instead of relying on one fixed context vector. This removes the information bottleneck of long sequences and substantially improves output quality.

How is Seq2Seq different from a classifier?

A classifier produces a fixed output such as a single label. Seq2Seq produces a variable-length output sequence whose length is decided during generation, which is what makes it suitable for translation, summarization and similar tasks.

Related methods

Sources

  1. Sutskever, I., Vinyals, O. & Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks. NeurIPS.

    Open publication reference
    Copied.
  2. Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H. & Bengio, Y. (2014). Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. EMNLP, 1724–1734.

    Open DOI reference
    Copied.