6 reports · 2012–2017

Training & data

How models are actually trained — objectives, scaling budgets, regularisation and the data decisions behind the results.

Fig. 02 / Field notesFollowing the gradient
Gradient descent on a quadratic loss surface. Successive steps converge toward the minimum.
Each step uses the slope of the same quadratic loss.

Dropout: Preventing Overfitting

Training & data · 5 min read · Beginner

Dropout randomly disables neurons during training, forcing redundant representations — a simple trick that helped make deep networks trainable.

dropoutregularizationtraining Read

Adam: Adaptive Moment Estimation

Training & data · 5 min read · Beginner

Adam combines momentum with per-parameter adaptive learning rates, making it the default optimizer for training deep neural networks.

adamoptimizationtraining Read

He Initialization: Delving Deep into Rectifiers

Training & data · 5 min read · Intermediate

He initialization scales starting weights for ReLU networks, keeping signal variance stable so very deep rectifier models can train.

initializationrelutraining Read

Layer Normalization

Training & data · 5 min read · Beginner

Layer Normalization normalizes across features instead of batches, stabilizing RNN and Transformer training at any batch size.

layer-normalizationnormalizationtransformers Read

mixup: Training on Convex Combinations

Training & data · 5 min read · Intermediate

mixup trains neural networks on convex combinations of examples and labels, improving generalization and robustness to noise.

mixupdata-augmentationregularization Read