AI-generated review 5 min read Intermediate 824 words

DETR: End-to-End Object Detection with Transformers

DETR replaces anchors and NMS with a transformer predicting objects as a set — end-to-end object detection with bipartite matching, explained simply.

object-detection detr transformers computer-vision end-to-end
Computer vision · 2020

Primary source: End-to-End Object Detection with Transformers (arXiv:2005.12872)

For years, object detectors were intricate pipelines: generate anchors, classify them, regress boxes, then prune thousands of duplicates with non-maximum suppression. In 2020, Nicolas Carion and colleagues at Facebook AI asked whether all that machinery could go. Their paper End-to-End Object Detection with Transformers (DETR) reframed detection as direct set prediction — a Transformer reads the image and emits the final boxes in one pass, with no anchors and no suppression step. It matched strong CNN detectors on COCO while being conceptually far simpler.

Why detection pipelines were so complicated

Two-stage detectors like Faster R-CNN and single-shot ones like YOLO share a design habit: predict far more boxes than there are objects, then clean up. Dense anchors blanket the image at many scales, most matching nothing, and overlapping duplicates survive until a hand-tuned suppression heuristic removes them. Every stage — anchor shapes, matching thresholds, suppression IoU — carries hyperparameters that interact in subtle ways. DETR's bet was that a sufficiently powerful sequence model could skip the over-generate-and-prune ritual entirely and map pixels to the final set of objects directly, learning the deduplication instead of hard-coding it.

From pixels to object sets with transformers

Image photo CNN backbone ResNet-50 features + pos 6 layers Encoder global context 6 layers + queries Decoder N = 100 slots Class head label or no-object Box head center, size
DETR: a CNN backbone extracts features, a transformer encoder adds global context, and the decoder turns 100 learned object queries into a set of class-plus-box predictions in parallel.

DETR's architecture has three parts. A standard CNN backbone (typically ResNet-50) converts the image into a compact feature map, flattened into a sequence with positional encodings. A Transformer encoder with 6 layers lets every location attend to every other, building global context — useful for separating overlapping instances. Then a 6-layer decoder takes N = 100 learned object queries, one per potential detection, and decodes them in parallel into 100 predictions, each a class label (or "no object") plus a box parameterized by center coordinates, height and width. Self-attention between queries is what suppresses duplicates: the slots learn to claim different objects, so no post-processing is needed. A small feed-forward network shared across all slots then produces each final class prediction and normalized box coordinates independently.

Bipartite matching: the loss that replaces NMS

The key invention is how these 100 unordered predictions are trained. DETR uses bipartite matching: the Hungarian algorithm finds the one-to-one assignment between predictions and ground-truth boxes that minimizes a cost combining class error and box error, L_{match} = -\log \hat{p}_{c} + L_{box}. Each ground-truth object is matched to exactly one prediction; all unmatched slots are supervised as "no object". The box loss itself mixes L1 distance with a scale-invariant generalized IoU term. Because training punishes any two slots for claiming the same object, the network learns unique assignments — duplicates simply never emerge, and non-maximum suppression becomes unnecessary by construction rather than by tuning.

COCO results, slow training and small objects

On COCO, DETR with a ResNet-50 backbone reached around 42 box AP, comparable to a well-tuned Faster R-CNN with a feature pyramid — remarkable for a first-of-its-kind design with no anchors, pyramids or suppression tuning. A longer-trained variant with a larger backbone pushed accuracy further, confirming that the architecture scales with capacity and schedule rather than depending on hand-crafted priors. It excelled at large objects, where global attention pays off. But two weaknesses stood out. Training took hundreds of epochs (around 10× longer than comparable detectors), partly because the attention must learn spatial structure from scratch. And small-object accuracy lagged, since the single low-resolution feature map loses fine detail. Follow-ups like Deformable DETR addressed both with multi-scale sparse attention, and the fixed 100-query design means crowded scenes beyond 100 instances need a larger configuration.

Why DETR changed detection design

DETR matters less as a leaderboard entry than as a proof that detection can be simple. It removed anchors, proposal heuristics and suppression tuning in one stroke, replacing them with a generic architecture shared with language models — the same attention operation, the same encoder-decoder stack. That generality unlocked extensions the pipeline era made awkward: panoptic segmentation from the same queries, 3D detection, and vision-language grounding where queries attend to text. Combined with backbones from the ViT family, the transformer-first detector is now a mainstream design, and DETR's set-prediction loss is the standard formulation new methods build on.

Bottom line

DETR turned detection into set prediction: a transformer decodes 100 learned queries into boxes, and Hungarian matching trains each object to be found exactly once. No anchors, no suppression — just attention, end to end.

Source

  • Carion et al., End-to-End Object Detection with Transformers, 2020. arXiv:2005.12872
  • Vaswani et al., Attention Is All You Need, 2017. arXiv:1706.03762
  • Ren et al., Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, 2015. arXiv:1506.01497

Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.

Cite this report
@misc{knowyourml-detr-end-to-end-object-detection
  title        = {DETR: End-to-End Object Detection with Transformers},
  author       = {{Know Your ML}},
  year         = {2020},
  howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=detr-end-to-end-object-detection}},
  note         = {AI-generated review of arXiv:2005.12872. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit — decision matrix, scorecards and trade-off analysis
Toolkit 7 decision tools Model-agnostic

Machine Learning Decision Toolkit

Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.

$24 View on Gumroad

References

Frequently asked questions

How does DETR detect objects without anchors?

DETR feeds CNN features into a transformer encoder-decoder and predicts a fixed set of objects in one pass. A bipartite matching loss assigns each prediction to at most one ground-truth box during training.

What is the Hungarian matching loss?

The Hungarian algorithm finds the one-to-one assignment between predictions and ground-truth boxes with the lowest total cost. This forces each object to be predicted exactly once, which is why DETR needs no non-maximum suppression.

What are DETR's main weaknesses?

DETR needs very long training schedules — hundreds of epochs — and performs worse on small objects than optimized CNN detectors. Follow-up work such as Deformable DETR largely fixed both issues.

What are object queries?

Object queries are a fixed set of learned positional embeddings fed to the decoder, one per potential detection. Each query specializes during training into detecting objects in certain positions or of certain sizes.

How to Remain Valuable When Intelligence Becomes Cheap — book cover
eBook 224 pages

How to Remain Valuable When Intelligence Becomes Cheap

This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?

Read the eBook