Segment Anything
How the Segment Anything project reframed segmentation as a promptable task and released a 1-billion-mask dataset that made zero-shot segmentation practical.
Primary source: Segment Anything (arXiv:2304.02643)
Ask a person to outline an object in a photo and the instruction is implicit: point at it and the boundary becomes obvious. Ask a segmentation network to do the same and the request has historically fallen outside its interface — classical segmentation models are built for a fixed vocabulary of categories, with a fixed number of output channels, trained on a specific dataset.
Segment Anything changed the interface. It defines promptable segmentation: any prompt that can specify what to segment — a point, a box, a rough mask, or a short phrase of text — should produce a valid mask. The task is deliberately open-ended, which is what makes a single model usable across many downstream segmentation problems.
Key takeaways
- The contribution is a task definition, not only a model: promptable segmentation accepts points, boxes, masks and text, and returns a mask.
- Ambiguity is handled by predicting three masks per prompt at different granularities, each with an estimated IoU score, instead of forcing one answer during training.
- The model splits into a heavyweight image encoder run once per image and a very light prompt encoder plus mask decoder, so interactive prompting is cheap after the initial pass.
- The accompanying SA-1B dataset contains over one billion masks across roughly eleven million images, collected with a model-in-the-loop data engine rather than pure human annotation.
- A single frozen model transfers zero-shot to unfamiliar segmentation tasks and datasets, which is the property that made it widely adopted as a component.
The architecture, and why it is split that way
SAM has three parts. A Vision Transformer image encoder, pretrained with masked autoencoding, converts the image into an embedding. A prompt encoder maps each prompt type into the same embedding space: points and boxes become positional encodings, masks are passed through convolutions, and text is handled by a CLIP text encoder. A small transformer mask decoder then attends from a set of learned output tokens to both embeddings and emits masks.
The asymmetry is the design decision that matters. Image encoding is expensive and prompt encoding is nearly free, so an application can encode a large image once and then answer many prompts against it interactively. The paper reports sub-second mask generation for a prompt given a precomputed image embedding, which turns segmentation into something closer to a click-driven tool than a batch job. Downstream systems exploit exactly this: encode once, prompt repeatedly.
Because prompts are ambiguous, the decoder outputs several candidate masks plus predicted quality scores. Ambiguity is therefore preserved rather than averaged away, and the caller decides which granularity is wanted. This is a small architectural choice with a large practical effect: one model can serve both "segment the whole object" and "segment the sub-part".
The data engine
The scale of SA-1B is the second half of the story. Building it required a data loop rather than a single annotation campaign, and the paper describes three stages: assisted-manual annotation, where people clicked on objects and refined masks with the model's help; a semi-automatic stage, where the model proposed masks for annotators to accept or reject; and a fully automatic stage, where a grid of prompt points was fed through the model and masks were filtered and deduplicated automatically.
Each stage both produced data and improved the model, which then made the next stage cheaper. The result was a dataset several orders of magnitude larger than typical segmentation datasets, with masks that are class-agnostic — they describe "an object" rather than "a dog" — which is precisely what a promptable model needs.
Zero-shot transfer
With a frozen model, the authors evaluate on tasks and datasets the model was not trained on, including edge detection, object proposals, instance segmentation and interactive segmentation. On many of these, SAM matches or exceeds prior fully supervised systems, and it does so with a single model rather than a task-specific architecture and training run.
That result is best read as engineering leverage rather than a claim of universal superiority. Some specialised models still win on their own benchmark, particularly where the category vocabulary or the domain (medical imaging, microscopy) differs sharply from natural images. But the cost of getting a reasonable segmentation for a new problem drops from "collect data and train" to "prompt the model".
Limitations and open questions
The paper is explicit about several weaknesses. Text-to-mask is the least developed prompt type: the model was not designed around language, and text prompts are weaker than geometric ones. Masks are class-agnostic and carry no semantic label, so a pipeline still needs something else to say what was segmented. The image encoder is heavy, which keeps a full encoder on-device out of reach for modest hardware. Fine structures such as hair or thin wires remain hard, and the model can inherit bias from the annotation process that produced SA-1B.
There are also open questions the architecture does not settle: whether promptable segmentation should be a separate model at all, how much of the gain comes from the data engine versus the task formulation, and how the licence and provenance of a dataset assembled from web images should be handled. Those questions have followed the model into the ecosystem built on top of it.
Further reading
- Segment Anything — the paper, the SA-1B dataset description and the zero-shot transfer results.
- Masked Autoencoders Are Scalable Vision Learners — the pretraining method used for SAM's image encoder.
- An Image is Worth 16x16 Words — the Vision Transformer that the encoder is built from.
- Learning Transferable Visual Models From Natural Language Supervision — CLIP, whose text encoder handles SAM's text prompts.
References
- Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., Dollár, P., & Girshick, R. (2023). Segment Anything. arXiv:2304.02643.
- He, K., Chen, X., Xie, S., Li, Y., Dollár, P., & Girshick, R. (2021). Masked Autoencoders Are Scalable Vision Learners. arXiv:2111.06377.
- Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv:2010.11929.
- Radford, A., Kim, J. W., Hallacy, C., et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020.
Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.
Cite this report
@misc{knowyourml-segment-anything
title = {Segment Anything},
author = {{Know Your ML}},
year = {2023},
howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=segment-anything}},
note = {AI-generated review of arXiv:2304.02643. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit
Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.
References
- Segment Anything (arXiv:2304.02643) — https://arxiv.org/abs/2304.02643
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (arXiv:2010.11929) — https://arxiv.org/abs/2010.11929
- Masked Autoencoders Are Scalable Vision Learners (arXiv:2111.06377) — https://arxiv.org/abs/2111.06377
- Learning Transferable Visual Models From Natural Language Supervision (arXiv:2103.00020) — https://arxiv.org/abs/2103.00020
Frequently asked questions
What is the Segment Anything Model (SAM)?
SAM is a promptable segmentation model: given an image and a prompt such as a point, a box, a coarse mask or free-form text, it returns a segmentation mask for the object the prompt refers to. It was trained on SA-1B, a dataset of more than one billion masks collected with a model-in-the-loop data engine, and it transfers to new segmentation tasks without task-specific fine-tuning.
Is SAM supervised or zero-shot?
It is trained with supervision on SA-1B, but it is used zero-shot: the same frozen model is applied to new datasets and prompt types without retraining. The published results show it matching or exceeding prior fully supervised systems on several benchmarks, while still trailing specialised models on some tasks.
Why does SAM output three masks per prompt?
Segmentation prompts are ambiguous: a point on a shirt could mean the shirt, the person or the group of people. SAM therefore predicts multiple masks at different levels of granularity together with an estimated IoU for each, and the ambiguity is resolved at use time rather than collapsed during training.
Part of
Beyond the paper
How to Remain Valuable When Intelligence Becomes Cheap
This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?