Vision Transformer
- Introduced by
- Dosovitskiy, A. et al.
- Year
- 2021
- Type
- Transformer architecture for images (self-attention over patches)
What does it do?
The Vision Transformer (ViT), introduced by Dosovitskiy and colleagues in 2021, splits an image into fixed-size patches, treats those patches as a sequence, and applies the Transformer self-attention mechanism to image classification.
Given enough training data, it surpasses convolutional neural networks (CNNs).
The intuition behind the method
Convolutional networks read an image through small local filters that slide across it. ViT takes a different view: it chops the image into a grid of equal-sized patches, lines them up like words in a sentence, and lets a Transformer decide which patches should pay attention to which. Because attention can directly relate any patch to any other, the model learns global relationships across the whole image rather than only local neighbourhoods.
History
The Vision Transformer was introduced by Dosovitskiy and colleagues in 2021 (An Image is Worth 16x16 Words), showing that the Transformer self-attention architecture, already dominant in language, could be applied directly to image patches and surpass CNNs when trained at scale. Follow-up work by Touvron and colleagues on data-efficient image transformers addressed ViT's appetite for data, making the approach more practical with smaller training budgets.
When to use it
Use ViT for image classification or prediction on continuous (pixel) image data when you have a large dataset — on the order of 1000 or more images, and ideally far more — and access to a GPU. It works best with a large pre-training corpus or transfer learning from a pre-trained model. On small image datasets (below a few hundred images) ViT cannot learn its patch-based attention reliably, and a CNN or classical machine-learning method such as Random Forest or SVM is the safer choice.
Limitations and considerations
- Requires a large training set (about 1000 images or more); on small data it underperforms CNNs.
- A GPU is required, and training from scratch is data- and compute-hungry.
- On very small datasets (below a few hundred images) patch-based attention fails to learn reliably.
- Strong results typically depend on a large pre-training corpus or transfer learning rather than training from scratch.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Surpasses CNNs on large image datasets by modelling global relationships through self-attention.
- Treats an image as a sequence of patches, applying the proven Transformer architecture to vision.
- Captures long-range dependencies across the whole image, not just local neighbourhoods.
- Benefits strongly from large-scale pre-training and transfer learning to pre-trained checkpoints.
- Does not assume normally distributed data.
Applications
- Image classification on large image datasets.
- Prediction tasks on continuous pixel data where global relationships matter.
- Transfer learning from large pre-trained image models to new vision tasks.
- Settings where modelling long-range dependencies across an image outperforms local convolutional filters.
How it works
How it works
The image is divided into fixed-size patches (for example 16x16 pixels). Each patch is flattened and linearly projected into an embedding vector, and the resulting embeddings form a sequence — the image analogue of a sequence of word tokens.
The patch sequence is fed through Transformer layers built on self-attention. Each patch can attend to every other patch, so the model captures long-range, global dependencies across the image rather than only local neighbourhoods, which is the core mechanism that lets ViT outperform CNNs on large datasets.
Formula
\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d_k}}\right)VA classification head reads the aggregated representation produced by the Transformer layers and outputs the predicted class. Because large pre-training matters so much here, transfer learning from a model pre-trained on a large image corpus is the usual route to strong performance.
Common pitfalls
- Training ViT from scratch on a small dataset instead of fine-tuning a pre-trained checkpoint.
- Expecting CNN-level accuracy without the large dataset ViT needs.
- Attempting to train without a GPU.
- Ignoring the failure thresholds — below a few hundred images, prefer a CNN or classical method.
- Overlooking transfer learning when pre-trained image models are available.
Frequently asked questions
How is ViT different from a CNN?
A CNN processes an image through local convolutional filters, while ViT splits the image into fixed-size patches, treats them as a sequence, and uses self-attention so any patch can relate to any other. This lets ViT model global relationships and, on large datasets, surpass CNNs.
How much data does ViT need?
ViT is data-hungry: it works best with around 1000 or more images and a large pre-training corpus. Below a few hundred images its patch-based attention cannot learn reliably, and a CNN or classical method is preferable.
Do I need a GPU?
Yes. ViT requires a GPU, and training it is data- and compute-intensive. Using a pre-trained checkpoint with transfer learning reduces the burden considerably.
What if my image dataset is small?
On small datasets ViT underperforms. Below roughly 500 images consider a method like Random Forest, and below a few hundred a CNN or SVM is the safer choice, or rely on transfer learning from a pre-trained model.
Related methods
Sources
Dosovitskiy, A. et al. (2021). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ICLR.
Open publication referenceCopied.Touvron, H. et al. (2021). Training Data-Efficient Image Transformers. ICML.
Open publication referenceCopied.