AI-generated review 5 min read Beginner 1,008 words

Layer Normalization

Layer Normalization normalizes across features instead of batches, stabilizing RNN and Transformer training at any batch size.

layer-normalization normalization transformers training rnn
Training & data · 2016

Primary source: Layer Normalization (arXiv:1607.06450)

Batch normalization transformed feed-forward and convolutional networks, but it has an awkward blind spot: it normalizes across the mini-batch, so anything that makes batches irregular — variable sequence lengths, tiny batch sizes, recurrent time steps — breaks its statistics. In 2016, Jimmy Lei Ba, Jamie Kiros, and Geoffrey Hinton proposed layer normalization in Layer Normalization: instead of normalizing each feature across the batch, normalize each example across its features. The change looks small, but it removed the dependence on batch size entirely and made the technique work identically at training and test time. A year later the Transformer adopted it for every sublayer, and through BERT and GPT-3 it became the normalization of the large-language-model era.

The core idea: normalize across features

Consider one training example passing through a layer with H hidden units, producing activations a_1, \dots, a_H. Layer normalization computes the mean and variance over those H units — the features of that single example — and normalizes with them:

\mu = \frac{1}{H}\sum_{i=1}^{H} a_i, \qquad \sigma^2 = \frac{1}{H}\sum_{i=1}^{H}(a_i - \mu)^2, \qquad \hat{a}_i = \frac{a_i - \mu}{\sqrt{\sigma^2 + \epsilon}}

As in batch normalization, learned per-feature gain g_i and bias b_i then rescale the normalized values via y_i = g_i \hat{a}_i + b_i, preserving the layer's expressive power. The crucial difference is the axis: batch normalization pools statistics over the batch dimension (m examples, one feature), while layer normalization pools over the feature dimension (one example, H features). Every example is normalized using only its own activations, so examples never interact and the operation is identical whether the batch holds one example or a thousand.

Why batches fail sequence models

Batch normalization assumes each feature's batch statistics are meaningful estimates of something stable. In recurrent networks that assumption collapses: different time steps have systematically different activation distributions, sequences in a batch have different lengths requiring padding, and test sequences may be longer than anything seen in training. Practitioners resorted to per-step statistics or gave up on normalization for RNNs entirely. Layer normalization sidesteps all of this because its statistics come from the current example alone — there is no batch to be irregular and no separate test-time behavior to maintain. The paper demonstrates the payoff directly: layer-normalized LSTMs train faster and reach better perplexity on language modeling, and converge in far fewer steps on tasks like handwritten-text recognition.

Batch norm: statistics down each column ex 1 ex 2 ex 3 μ, σ² over batch Layer norm: statistics across each row ex 1 ex 2 ex 3 μ, σ² over features rows = examples · columns = features
Normalization axes compared: batch normalization pools each feature's statistics down a column of examples, while layer normalization pools each example's statistics across its row of features.

The invariance that stabilizes training

Layer normalization has a clean geometric property the paper emphasizes: rescaling all of a layer's weights by the same factor leaves the normalized output — and therefore the loss — unchanged. If every weight doubles, every activation doubles, but the mean and variance double along with them and the normalized values come out identical. This scale invariance means the optimizer cannot wander off by inflating weights, and it makes the effective learning dynamics far more stable: growing the weights implicitly shrinks the learning rate rather than destabilizing the forward pass. Batch normalization shares a similar invariance, but layer normalization achieves it without any dependence on other examples, which is what makes it safe for recurrent and online settings where batches are meaningless.

Layer normalization in the Transformer

The Transformer's 2017 debut made layer normalization famous. In the original architecture, every attention sublayer and every feed-forward sublayer sits inside a residual connection followed by layer normalization — Add & Norm, in the paper's diagram. With per-token representations of dimension 512, the mean and variance are computed over each token's 512 features independently, so padding tokens and variable lengths cause no trouble. Later variants moved the normalization inside the residual branch before the sublayer (pre-norm), which further stabilizes very deep stacks and is now the standard layout in large language models. ResNet-style vision models kept batch normalization, but essentially every sequence model since 2017 — translators, speech recognizers, and the GPT lineage — normalizes by layer.

Limitations and modern variants

Layer normalization is not free of trade-offs. Because it adds no batch-dependent noise, it provides none of batch normalization's incidental regularization, so models still need Dropout, weight decay, or augmentation. It also assumes the layer has enough features for stable statistics — with very narrow layers the estimates get noisy. And in convolutional vision models with large batches, batch normalization typically still wins on accuracy. Research has produced lighter variants: RMSNorm drops the mean-centering and normalizes by the root-mean-square alone, saving compute with little quality loss, and is now common in large language models. The practical rule is straightforward: normalize by layer (or RMS) for sequences, Transformers, and small batches, and by batch for convolutional vision with batches of reasonable size.

What the paper showed

Ba, Kiros, and Hinton validated layer normalization across recurrent and feed-forward tasks: word-level language modeling with LSTMs, image classification with batch-size-one training where batch normalization cannot even be applied, and sequence tasks with long-range structure. Layer-normalized recurrent networks consistently converged faster than unnormalized baselines and matched or beat batch-normalized feed-forward networks where both applied. The paper also showed that the technique combines cleanly with other stabilizers of its era. Its lasting contribution, though, is architectural: by freeing normalization from the batch axis, it supplied the missing ingredient that let attention-based sequence models scale to the depths behind today's language models.

Source

  • Ba et al., Layer Normalization, 2016. arXiv:1607.06450
  • Ioffe & Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, 2015. arXiv:1502.03167
  • Vaswani et al., Attention Is All You Need, 2017. arXiv:1706.03762

Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.

Cite this report
@misc{knowyourml-layer-normalization-sequence-training
  title        = {Layer Normalization},
  author       = {{Know Your ML}},
  year         = {2016},
  howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=layer-normalization-sequence-training}},
  note         = {AI-generated review of arXiv:1607.06450. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit — decision matrix, scorecards and trade-off analysis
Toolkit 7 decision tools Model-agnostic

Machine Learning Decision Toolkit

Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.

$24 View on Gumroad

References

Frequently asked questions

How is layer normalization different from batch normalization?

Batch normalization computes statistics across the mini-batch for each feature, while layer normalization computes statistics across the features for each example. Layer normalization is therefore independent of batch size and works identically at training and test time.

Why is layer normalization better for recurrent networks?

Recurrent networks process sequences of varying length, so batch statistics change from step to step and cannot be shared cleanly. Normalizing per example across the layer's features avoids the problem entirely.

Where is layer normalization used in the Transformer?

In the original Transformer every sublayer output passes through layer normalization inside a residual connection. Nearly all later Transformer variants keep this pattern, sometimes moving the normalization before the sublayer (pre-norm) for extra stability.

Does layer normalization regularize the model?

Yes. Unlike batch normalization it adds no batch-dependent noise, so it provides no regularization effect of its own. Models using it typically still rely on Dropout, weight decay, or data augmentation.

How to Remain Valuable When Intelligence Becomes Cheap — book cover
eBook 224 pages

How to Remain Valuable When Intelligence Becomes Cheap

This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?

Read the eBook