Skip to content
Back to search results

Longformer / BigBird

Introduced by
Beltagy, Peters & Cohan (Longformer); Zaheer et al. (BigBird)
Year
2020
Type
Sparse-attention Transformer for long sequences
Font size

What does it do?

Long-sequence Transformers such as Longformer (Beltagy, Peters & Cohan, 2020) and BigBird (Zaheer et al., 2020) replace the standard Transformer's O(n²) attention with sparse attention patterns that scale linearly, O(n), with sequence length.

This lets a single model attend over thousands of tokens — full documents, legal texts, or genomic sequences — that would not fit a conventional Transformer.

The intuition behind the method

A standard Transformer compares every token with every other token, so doubling the text roughly quadruples the work — which is why most models cap out around 512 tokens. Long-sequence Transformers notice that most useful relationships are local, so each token mainly looks at its near neighbours through a sliding window, while a few designated tokens are allowed to look at everything. This keeps the cost growing in step with the text length instead of exploding, so very long documents become tractable.

History

Both methods appeared in 2020 as responses to the quadratic attention bottleneck that limited Transformers to short inputs. Longformer (Beltagy, Peters & Cohan) introduced sliding-window plus global attention for long documents, while BigBird (Zaheer et al., NeurIPS) added random attention to its sparse pattern and provided theoretical results on its expressiveness. Together they established sparse attention as the standard route to processing long sequences.

When to use it

Use long-sequence Transformers when you have long-document text (typically beyond 512 tokens) and need classification or explanation over the full document — long reports, legal texts, or genomic sequences. They assume genuinely long inputs are available, a GPU is strongly recommended, and global token positions must be chosen sensibly. They need a reasonable amount of data: below about 500 documents the model tends to overfit, and below about 100 documents training is not meaningful and classical ML such as Random Forest or XGBoost is preferable.

Limitations and considerations

  • A GPU is effectively required; training and inference are computationally heavy.
  • Overfits on small document sets — below about 500 documents results are unreliable.
  • Below about 100 documents training is meaningless and classical ML should be used instead.
  • Global token positions must be chosen correctly, or long-range information flow degrades.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Processes thousands of tokens — whole documents — that exceed a standard Transformer's window.
  • Linear O(n) attention scaling instead of the quadratic O(n²) cost of dense attention.
  • Combines local sliding-window context with global tokens for long-range information flow.
  • Well suited to long-document classification in law, genomics, and long-form text.

Applications

  • Classification of long documents such as reports and articles.
  • Analysis of legal texts that exceed standard Transformer windows.
  • Modelling long genomic sequences.
  • Explanation and classification tasks over full-length text rather than truncated excerpts.

How it works

How it works

1

The core move is to drop the dense, all-pairs attention of the standard Transformer — which costs O(n²) in the sequence length n — and approximate it with a sparse pattern that costs only O(n). Because attention is the bottleneck, this single change is what unlocks inputs of thousands of tokens.

Formula
O(n^2) \;\rightarrow\; O(n)
2

Most attention is restricted to a sliding window around each token, so a token attends to a fixed band of nearby tokens rather than the entire sequence. Local context — the words on either side — carries most of the signal, and computing only this band is what makes the cost grow linearly with length.

3

A small set of designated global tokens are allowed to attend to, and be attended by, the whole sequence, giving the model a path for long-range information to flow across the document. Choosing these global token positions correctly (for example a classification token or query tokens) is important, since they are the bridge between distant parts of the text.

4

On top of the sparse-attention encoder, a task head produces the final result — for example a document-level classification or an explanatory prediction over the long input. The result is a model that can process full-length documents that a 512-token Transformer would have to truncate.

Common pitfalls

  • Applying it to short texts that fit a standard 512-token Transformer, paying the complexity for no benefit.
  • Choosing global token positions carelessly, breaking the model's long-range pathways.
  • Training on a small document set (n below a few hundred) and trusting the overfit result.
  • Attempting to train without a GPU and giving up on a workable model.

Frequently asked questions

How is this different from a standard Transformer like BERT?

A standard Transformer uses dense attention, comparing every token with every other token at O(n²) cost, which in practice caps inputs around 512 tokens. Longformer and BigBird use sparse attention — mostly local sliding windows plus a few global tokens — so cost grows linearly and inputs of several thousand tokens become feasible.

When should I prefer Longformer or BigBird over classical ML?

Use them when your text is genuinely long (beyond 512 tokens) and you have enough documents. With fewer than about 500 documents the model tends to overfit, and below roughly 100 documents training is not meaningful — classical methods such as Random Forest or XGBoost are the safer choice there.

What are global tokens and why do they matter?

Global tokens are a small set of positions allowed to attend to, and be attended by, the entire sequence. They are the channel through which long-range information moves across the document, so choosing their positions correctly is important for performance.

Do I need special hardware?

A GPU is strongly recommended. These models still process long sequences with sizeable networks, so training and inference are computationally demanding even with the linear-scaling attention.

Related methods

Sources

  1. Beltagy, I., Peters, M. E. & Cohan, A. (2020). Longformer: The Long-Document Transformer. arXiv.

    Open publication reference
    Copied.
  2. Zaheer, M. et al. (2020). Big Bird: Transformers for Longer Sequences. NeurIPS.

    Open publication reference
    Copied.