AI-generated review 5 min read Intermediate 865 words

Masked Autoencoders Are Scalable Vision Learners

Masked Autoencoders pretrain Vision Transformers by reconstructing 75% masked patches, scaling self-supervised vision with simple asymmetric design.

masked-autoencoders mae self-supervised vision-transformers pretraining
Computer vision · 2021

Primary source: Masked Autoencoders Are Scalable Vision Learners (arXiv:2111.06377)

In language modeling, BERT showed that masking words and predicting them back teaches deep, transferable representations. Vision researchers tried the analogous trick for years — mask image regions, reconstruct pixels — but the results consistently underwhelmed. Images are highly redundant: hiding a small patch lets the network cheat by interpolating neighboring pixels, learning local texture instead of global understanding.

The 2021 paper Masked Autoencoders Are Scalable Vision Learners by He, Chen, Xie, Li, Dollár, and Girshick finally made masked pretraining work for vision. MAE combines an extreme 75% masking ratio with an asymmetric encoder-decoder design on top of Vision Transformers, producing a method that is simple, fast, and scales beautifully to large models.

The core idea

MAE treats an image as a grid of non-overlapping patches, exactly as ViT does, then randomly masks out 75% of them. Only the remaining 25% visible patches are fed to the encoder; a lightweight decoder then receives the encoded patches plus learned mask tokens and must reconstruct the pixel values of the missing patches. The loss is mean squared error computed on masked patches only.

The high masking ratio is the whole game. With only a quarter of the image visible, local interpolation fails — the missing regions are too large — so the network must reason about objects, scenes, and composition to guess what goes where. The paper's ablations show accuracy climbing steadily as masking rises from 40% to 75%, then dropping past 85% when too little signal remains. The sweet spot is far above the 15% masking used in language.

The asymmetric design

Patchify + mask 75% 2 of 9 visible Large encoder ViT on visible only Small decoder encoded + mask tokens Target reconstruct masked pixels Loss: MSE on masked patches only · decoder discarded after pretraining
MAE pretraining: most patches are masked, the large encoder sees only visible patches, and a small decoder reconstructs the missing pixels; only masked patches contribute to the loss.

The architecture is deliberately lopsided. The encoder is a full-size ViT, but it processes only the 25% visible patches — no mask tokens, no wasted compute on holes. Because self-attention scales quadratically with sequence length, this sparsity cuts encoder FLOPs by roughly 4x and memory even more. The decoder is narrow and shallow (around 8 blocks with 512 dimensions in the paper), just enough to solve the pixel-prediction task, and it is discarded entirely after pretraining.

This asymmetry also shapes what is learned. Reconstruction specialization lives in the disposable decoder, while the encoder is free to build abstract representations useful for downstream tasks. The paper shows that a too-powerful decoder actually hurts transfer, since the encoder can offload work onto it.

Reconstruction targets

Predicting raw pixels might seem like a weak objective — pixels carry lighting, compression artifacts, and other nuisances. The paper's fix is simple and effective: normalize each patch's pixels to zero mean and unit variance before computing the loss. This per-patch normalization emphasizes contrast and structure within each patch rather than absolute brightness, and it improves fine-tuning accuracy by a clear margin.

Alternatives like tokenized targets (as in BEiT, which predicts discrete visual tokens from a pretrained tokenizer) perform similarly but require a separate pretrained tokenizer, adding complexity. MAE's normalized-pixel regression needs no tokenizer, no momentum encoder, and no negative samples — just patches, masks, and mean squared error.

What the paper showed

Pretraining a ViT-Huge with MAE for 1600 epochs and fine-tuning on ImageNet reached around 87.8% top-1 accuracy, the best reported at the time for methods using only ImageNet data. Gains grew with model size: larger ViTs benefited more, confirming the "scalable" in the title. Transfer results were equally strong, with MAE-pretrained models beating supervised pretraining on COCO detection and ADE20K segmentation. Training was also fast — roughly 3x faster per epoch than contrastive counterparts like MoCo v3, thanks to encoding only visible patches.

The reconstructions themselves told the story qualitatively: even with 75% masked, MAE plausibly completed objects and scenes it had never seen labeled, suggesting genuine visual understanding rather than texture matching.

Legacy and limitations

MAE established masked modeling as the second pillar of visual self-supervision alongside contrastive learning, and its lineage — data2vec, VideoMAE, MultiMAE, and masked pretraining for CLIP-style models — extends across modalities. Compared with SimCLR-style contrastive methods, MAE needs no large batches, no careful augmentation design, and far less memory, making it accessible on modest hardware.

Limitations are honest ones. Linear probing — freezing the encoder and training only a classifier — lags contrastive methods noticeably, because reconstruction features are less linearly separable; MAE shines under fine-tuning, not frozen evaluation. Pixel reconstruction also spends capacity on high-frequency details irrelevant to recognition, which motivated later feature-prediction variants like I-JEPA. Still, as a scalable, dead-simple recipe for pretraining large vision models on unlabeled images, MAE set the standard that successors still chase.

Source

  • He et al., Masked Autoencoders Are Scalable Vision Learners, 2021. arXiv:2111.06377
  • Dosovitskiy et al., An Image is Worth 16x16 Words, 2020. arXiv:2010.11929
  • Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers, 2018. arXiv:1810.04805

Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.

Cite this report
@misc{knowyourml-mae-masked-autoencoders-scalable-vision
  title        = {Masked Autoencoders Are Scalable Vision Learners},
  author       = {{Know Your ML}},
  year         = {2021},
  howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=mae-masked-autoencoders-scalable-vision}},
  note         = {AI-generated review of arXiv:2111.06377. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit — decision matrix, scorecards and trade-off analysis
Toolkit 7 decision tools Model-agnostic

Machine Learning Decision Toolkit

Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.

$24 View on Gumroad

References

Frequently asked questions

Why does MAE mask such a high fraction of patches?

Masking 75% of patches removes most redundancy, so the network cannot solve the task by copying nearby pixels. It must learn global object structure and semantics to fill in the large gaps.

How is MAE computationally efficient?

Only the 25% visible patches pass through the large encoder, which cuts compute dramatically. The lightweight decoder then reconstructs all patches from the encoded visible ones plus mask tokens.

How does MAE differ from BERT?

BERT masks about 15% of discrete word tokens and predicts them with a classification loss. MAE masks 75% of continuous image patches and reconstructs normalized pixel values with regression.

What happens to the decoder after pretraining?

After pretraining the decoder is discarded and the encoder is fine-tuned on labeled data like a standard ViT. The pretrained weights provide a far better starting point than random initialization.

How to Remain Valuable When Intelligence Becomes Cheap — book cover
eBook 224 pages

How to Remain Valuable When Intelligence Becomes Cheap

This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?

Read the eBook