Dropout: Preventing Overfitting
Dropout randomly disables neurons during training, forcing redundant representations — a simple trick that helped make deep networks trainable.
Primary source: Dropout: A Simple Way to Prevent Neural Networks from Overfitting (arXiv:1207.0580)
Deep networks have millions of parameters and a bad habit: memorizing the training set instead of learning patterns that generalize. The 2012 dropout paper proposed a fix of startling simplicity — during training, randomly switch off half the neurons on every batch. No unit can rely on any other being present, so each must learn features that are useful on their own.
How it works
On each training step, every unit (except outputs) is kept with probability p — 0.5 for hidden layers is the classic default — and the survivors are scaled up by 1/(1−p) so expected activations stay constant. At test time, all units stay on and no scaling is applied. This "inverted" form means the deployed model is just the plain network.
The effect is an implicit ensemble. Each training batch samples a different thinned sub-network from 2^n possibilities; they share weights, so training one updates all of them. At test time, using the full network approximates averaging the whole ensemble — the cheap version of a technique that would otherwise cost thousands of models.
What the results showed
Dropout cut error rates across MNIST, speech, and document classification, and — decisively — it was a key ingredient in AlexNet's 2012 ImageNet win. Combined with ReLU and GPU training, it showed that very large networks could generalize if regularized aggressively. The paper also introduced max-norm constraints and the practical recipe (higher learning rates, momentum) that made dropout train well.
Why the ensemble view holds up
The test-time approximation is not exact — averaging weights is not averaging predictions for nonlinear networks — but it works because dropout also changes what gets learned. Co-adaptation is the enemy: without dropout, a unit can rely on a partner to correct its mistakes, producing brittle feature pairs that shatter on new data. With half the network missing each batch, every unit must be independently useful, which pushes representations toward redundancy and linear separability.
There is also a Bayesian reading (Gal & Ghahramani, 2016): dropout training approximates variational inference in a deep Gaussian process, and leaving dropout on at test time with multiple forward passes ("MC dropout") yields genuine uncertainty estimates. It remains one of the cheapest uncertainty methods available — no extra parameters, just repeated sampling.
Dropout's descendants
The idea outgrew its original form. DropConnect drops weights instead of activations. Stochastic Depth drops entire residual layers during training, effectively training an ensemble of networks at different depths. DropPath and LayerDrop do the same for transformer blocks. In transformers specifically, attention dropout (on the weights, not the values) and embedding dropout are standard — GPT, BERT, and LLaMA all train with them.
The throughline: noise injected during training, removed at test time, forces robustness. Data augmentation is the same principle applied to inputs; label smoothing applies it to targets.
Where it stands now
Modern training leans more on batch normalization, augmentation, weight decay, and sheer data scale, so dropout is less universal than in 2014. But it never left: transformer FFN layers, attention dropout, embedding dropout, and classifier heads still use it routinely. Its descendants — DropConnect, Stochastic Depth, DropPath — apply the same idea to weights, layers, and residual paths.
Dropout versus batch normalization
The two great regularizers have an uneasy relationship. Batch normalization reduces internal covariate shift and adds its own regularization noise, which in CNNs largely replaced dropout in convolutional layers. But they can conflict: dropout's random zeros shift batch statistics, making training noisier than either alone — the "variance shift" problem that led most vision architectures to use batch norm in conv layers and dropout only in the final fully-connected head.
Transformers sidestepped the conflict by using layer normalization (per-example, no batch statistics) plus modest dropout everywhere: typically 0.1 on attention weights, FFN activations, and residuals. The modern default is small dropout in many places rather than 0.5 in one — the same principle, spread thin.
Bottom line
Dropout regularizes by deleting: random units off per batch, forcing every neuron to earn its place. One line of code, no extra parameters, and a central reason deep networks generalize at all.
Sources
- Srivastava et al., Dropout: A Simple Way to Prevent Neural Networks from Overfitting, 2014. arXiv:1207.0580
Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.
Cite this report
@misc{knowyourml-dropout-preventing-overfitting
title = {Dropout: Preventing Overfitting},
author = {{Know Your ML}},
year = {2012},
howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=dropout-preventing-overfitting}},
note = {AI-generated review of arXiv:1207.0580. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit
Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.
References
- Dropout: A Simple Way to Prevent Neural Networks from Overfitting (arXiv:1207.0580) — https://arxiv.org/abs/1207.0580
Frequently asked questions
How is dropout implemented?
During training each unit is zeroed with probability p (often 0.5 for hidden layers) and survivors are scaled by 1/(1-p). At test time all units stay on with no scaling — inverted dropout.
Is dropout still used?
Less than before. Batch normalization, large datasets, and augmentation cover much of its job, but dropout still appears in fully-connected heads, embeddings, and attention layers of many models.
Beyond the paper
How to Remain Valuable When Intelligence Becomes Cheap
This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?