Dropout: Preventing Overfitting
Dropout randomly disables neurons during training, forcing redundant representations — a simple trick that helped make deep networks trainable.
6 reports · 2012–2017
How models are actually trained — objectives, scaling budgets, regularisation and the data decisions behind the results.
Dropout randomly disables neurons during training, forcing redundant representations — a simple trick that helped make deep networks trainable.
Adam combines momentum with per-parameter adaptive learning rates, making it the default optimizer for training deep neural networks.
Batch Normalization normalizes layer inputs per mini-batch, enabling higher learning rates and faster, more stable deep network training.
He initialization scales starting weights for ReLU networks, keeping signal variance stable so very deep rectifier models can train.
Layer Normalization normalizes across features instead of batches, stabilizing RNN and Transformer training at any batch size.
mixup trains neural networks on convex combinations of examples and labels, improving generalization and robustness to noise.