AI-generated review 7 min read Intermediate 526 words

An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale

The Vision Transformer applied a plain transformer to image patches and matched CNNs at scale — proving convolutions were optional for recognition.

vit transformers computer-vision attention scaling
Computer vision · 2020

Primary source: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (arXiv:2010.11929)

For half a decade, convolutions seemed essential to vision: they encode translation invariance and locality, and every successful architecture used them. The Vision Transformer (ViT), published by Google Research in 2020, challenged that assumption directly: treat an image as a sequence of patches — like words in a sentence — and run a standard transformer encoder on it. With enough data, ViT matched or beat the best CNNs, and it did so without any convolution at all.

The core idea

Split an image into fixed-size patches (e.g., 16×16 pixels), flatten each patch into a vector, add a learned position embedding, and feed the sequence to a standard transformer encoder — the same architecture used for text. A special [class] token's output, after the encoder, feeds a classification head.

The bet was that attention is a general-purpose computation and that the inductive biases of convolutions (locality, weight sharing) are useful mostly when data is scarce. Given enough data, a transformer can learn those biases itself — and can exploit long-range relationships that convolutions capture only with great difficulty.

How it works

  • Patch embedding: a linear projection maps each 16×16×3 patch to a 768-dim embedding (a 14×14 grid of patches for a 224×224 input = 196 tokens, plus the [class] token).
  • Transformer encoder: standard multi-head self-attention + MLP blocks with LayerNorm and residual connections, identical to BERT's encoder.
  • Position embeddings: learned 1D embeddings added to each patch token (the paper shows 1D works as well as 2D).
  • Scaling: ViT-Large/16 and ViT-Huge/14 (632M parameters) trained on ImageNet-21k (14M images) or JFT-300M (300M images).
  • Results: pre-trained on JFT-300M, ViT-Huge reached 88.55% top-1 on ImageNet — better than the concurrent SOTA CNN (BiT-L, 87.54%). On smaller datasets (ImageNet-1k only), ViT lagged behind ResNets of similar compute, confirming that the architecture needs scale.

Why it matters

  • The unification of vision and language. ViT showed the same transformer block could dominate both modalities, clearing the path for multimodal models and, later, unified vision-language transformers.
  • Scaling behavior. Vision transformers improve smoothly with model and data size — better scaling curves than CNNs — which is exactly what the compute-rich era needed.
  • Simple, reusable design. ViT's patch embedding + transformer is far simpler to implement than a modern CNN stack and inherits all transformer tooling (pre-training, distillation, efficient attention).
  • Ecosystem. ViT spawned an enormous family: DeiT (data-efficient training), Swin (hierarchical windows), DINO (self-supervised ViT), and the vision encoders inside CLIP, DALL·E, and every modern multimodal system.

Limitations and open questions

ViT lacks the built-in inductive biases of convolutions, so it underperforms CNNs when trained on small datasets — the paper needed JFT-300M or ImageNet-21k pre-training to shine. Self-attention is quadratic in the number of patches, making high-resolution inputs expensive (motivating Swin's windowed attention). The paper also raised questions about how much of vision truly needs long-range interactions versus local structure, which windowed and hybrid architectures continue to explore.

Key takeaways

  • ViT = image as patch sequence → standard transformer encoder → classification head.
  • With large-scale pre-training (300M images), ViT-Huge beat the best CNNs on ImageNet.
  • On small datasets CNNs still win — transformers need data or strong augmentation to compete.
  • ViT unified vision and language architectures and became the default vision backbone of the multimodal era.

Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.

Cite this report
@misc{knowyourml-vit-an-image-is-worth-16x16-words
  title        = {An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale},
  author       = {{Know Your ML}},
  year         = {2020},
  howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=vit-an-image-is-worth-16x16-words}},
  note         = {AI-generated review of arXiv:2010.11929. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit — decision matrix, scorecards and trade-off analysis
Toolkit 7 decision tools Model-agnostic

Machine Learning Decision Toolkit

Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.

$24 View on Gumroad

References

How to Remain Valuable When Intelligence Becomes Cheap — book cover
eBook 224 pages

How to Remain Valuable When Intelligence Becomes Cheap

This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?

Read the eBook