He Initialization: Delving Deep into Rectifiers
He initialization scales starting weights for ReLU networks, keeping signal variance stable so very deep rectifier models can train.
Primary source: Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (arXiv:1502.01852)
Before a network has learned anything, its weights must start somewhere — and that starting point matters enormously. Initialize too small and signals fade to nothing as they pass through dozens of layers; too large and they explode. The standard prescription through 2014 was Xavier (Glorot) initialization, designed for sigmoid and tanh networks. But rectifier networks — built on ReLU, the activation behind the AlexNet breakthrough — were quietly violating Xavier's assumptions. In 2015, Kaiming He and colleagues fixed this in Delving Deep into Rectifiers, deriving an initialization scaled specifically for ReLU. That derivation, He initialization, became the default for rectifier networks and a prerequisite for the very deep models that followed, including ResNet from largely the same team.
Why initialization decides trainability
Each layer multiplies its inputs by a weight matrix: get the weight scale wrong and the damage compounds exponentially with depth. If the variance of activations shrinks by a factor at every layer, a 20-layer network receives only noise at its top; gradients flowing backward suffer the mirror-image fate, vanishing or exploding on their way down. Xavier initialization solved this for symmetric, roughly linear activations by setting the weight variance to 1/n, where n is the number of incoming connections (fan-in) — a scale at which variance is preserved in both directions. But ReLU is neither symmetric nor linear: it zeroes out every negative input. A rectifier layer initialized à la Xavier therefore discards roughly half its signal variance at every step, and deep stacks simply stop learning — the paper reports a 30-layer rectifier network that fails to converge at all.
Deriving the factor of two
He et al. repeat the variance analysis with ReLU's truncation taken into account. Assume the inputs to a layer are symmetric around zero with variance \text{Var}(x), and the weights are independent zero-mean draws with variance \text{Var}(w). The pre-activation of one unit sums n products w_i x_i, so its variance is n \cdot \text{Var}(w) \cdot \text{Var}(x). ReLU then zeroes the negative half — for a symmetric distribution this halves the variance — giving output variance \tfrac{1}{2} n \cdot \text{Var}(w) \cdot \text{Var}(x). To preserve variance across the layer (\text{Var}(y) = \text{Var}(x)), the weights need:
The factor of 2 is exactly the compensation for ReLU discarding half its input. The same analysis run backward yields 2/\hat{n} with fan-out \hat{n}; in practice the fan-in form is the standard choice for hidden layers.
Gaussian versus uniform forms
In practice the variance prescription 2/n can be realized by either distribution family. The Gaussian form draws weights from a zero-mean normal distribution with standard deviation \sqrt{2/n}; the uniform form draws from [-\sqrt{6/n}, +\sqrt{6/n}], whose variance is also 2/n. Deep learning frameworks expose both — PyTorch's kaiming_normal_ and kaiming_uniform_ — and the two perform nearly interchangeably. For convolutional layers, n is the fan-in: kernel height × kernel width × input channels, since every output unit sees that many incoming connections. Biases are simply initialized to zero. The derivation assumes independent inputs and weights, which holds exactly once — before training correlates them — but that is all the theory needs to get the first passes into a healthy range.
PReLU and the road to human-level ImageNet
The paper's second contribution is the Parametric ReLU (PReLU), which replaces ReLU's flat zero for negative inputs with a learnable slope a_i, i.e. f(y_i) = \max(0, y_i) + a_i \min(0, y_i). The slopes are learned by backpropagation at negligible extra cost. Armed with PReLU and the new initialization, the authors trained deep rectifier networks to 4.94% top-5 error on ImageNet — the first result reported to surpass the roughly 5.1% human-level error on the same task. Modern networks mostly kept the initialization and dropped PReLU in favor of plain ReLU, GELU, or SiLU variants.
He initialization in the modern stack
He initialization is now the quiet default underneath deep learning: the standard starting point for convolutional networks, vision Transformers' MLP blocks, and residual architectures everywhere. It composes cleanly with the other stabilizers of its era — batch normalization reduces sensitivity to initialization, but even normalized networks train better from a well-scaled start, and normalizer-free designs depend on it critically. A related technique from the same period, Dropout, interacts only mildly with the choice of starting scale. Practical rules: use He (kaiming) initialization for ReLU-family activations, Xavier for tanh/sigmoid and linear layers, and specialized schemes (small scaled normals, zeroed residual branches) for very deep Transformers. When a from-scratch model refuses to train, the initialization scale remains one of the first suspects.
When the assumptions break
The 2/n derivation leans on idealized conditions — independent zero-mean weights and symmetric inputs — that real training quickly violates. With normalization layers, activations are rescaled every step, which dulls the importance of the starting scale (though a good start still speeds early training). Exotic activations need their own analysis: SELU networks want LeCun initialization (1/n) to preserve their self-normalizing property, and quantized or sparse networks may need custom scales. Depth also matters — even perfectly preserved variance does not guarantee trainability past hundreds of layers without residual connections. Treat He initialization as the correct default for rectifier networks rather than a universal law: match the scale to the activation, and revisit it whenever the architecture departs from the standard recipe.
Source
- He et al., Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, 2015. arXiv:1502.01852
- He et al., Deep Residual Learning for Image Recognition, 2015. arXiv:1512.03385
- Ioffe & Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, 2015. arXiv:1502.03167
Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.
Cite this report
@misc{knowyourml-he-initialization-delving-into-rectifiers
title = {He Initialization: Delving Deep into Rectifiers},
author = {{Know Your ML}},
year = {2015},
howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=he-initialization-delving-into-rectifiers}},
note = {AI-generated review of arXiv:1502.01852. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit
Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.
References
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (arXiv:1502.01852) — https://arxiv.org/abs/1502.01852
- Deep Residual Learning for Image Recognition (arXiv:1512.03385) — https://arxiv.org/abs/1512.03385
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift (arXiv:1502.03167) — https://arxiv.org/abs/1502.03167
Frequently asked questions
Why does Xavier initialization fail for ReLU networks?
Xavier initialization assumes roughly linear activations and preserves variance in both directions under that assumption. ReLU kills about half of its inputs, so Xavier-started rectifier networks lose roughly half their signal variance per layer.
Where does the factor of 2 in He initialization come from?
The factor of 2 compensates for ReLU discarding the negative half of its input. With weights drawn at variance 2/n, the surviving positive half carries the same total variance the layer received.
Should you use the fan-in or fan-out form?
Use the fan-in form, variance 2/n_in, for most feed-forward and convolutional layers. The fan-out form matters mainly for the final layer, where preserving backward gradient variance helps the last layer learn.
Is He initialization related to ResNet?
Yes. The paper's initialization was a prerequisite for training the very deep rectifier networks that followed, including the 152-layer ResNet, whose authors are largely the same team.
Beyond the paper
How to Remain Valuable When Intelligence Becomes Cheap
This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?