AI-generated review 5 min read Beginner 1,005 words

Adam: Adaptive Moment Estimation

Adam combines momentum with per-parameter adaptive learning rates, making it the default optimizer for training deep neural networks.

adam optimization training gradient-descent adaptive-learning-rates
Training & data · 2014

Primary source: Adam: A Method for Stochastic Optimization (arXiv:1412.6980)

Every neural network is trained by an optimizer — the algorithm that turns gradients into weight updates. For years the default was stochastic gradient descent with hand-tuned learning rates and momentum, which demanded careful babysitting for each new model. In 2014, Diederik Kingma and Jimmy Ba proposed Adam, short for adaptive moment estimation, in Adam: A Method for Stochastic Optimization. Adam gives every parameter its own adaptive learning rate by combining two older ideas, momentum and RMSProp-style scaling, into one update rule with sensible defaults. It worked so well out of the box that it became the default optimizer across deep learning, from the convolutional networks of the AlexNet era to BERT and GPT-3.

The core idea: momentum meets adaptive rates

Two extensions to plain gradient descent each fix a different weakness. Momentum keeps a moving average of past gradients and steps in that averaged direction, which accelerates progress along consistent slopes and damps oscillation across narrow valleys. RMSProp-style methods instead track the average squared gradient per parameter and divide each update by its square root, so parameters with large, frequent gradients take smaller steps while quiet parameters take larger ones. Adam simply keeps both moving averages at once: the first moment m_t estimates the mean gradient (momentum), and the second moment v_t estimates the uncentered variance (adaptive scaling). Each parameter update is then the momentum direction rescaled by its own historical magnitude — fast along steady directions, cautious where gradients swing wildly.

The Adam update rule

At step t with gradient g_t, Adam updates its two exponential moving averages with decay rates \beta_1 and \beta_2:

m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t, \qquad v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2

Because both averages start at zero, they are biased toward zero early in training. Adam corrects this with factors that grow from small values toward one:

\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \qquad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}

The parameters are then updated with step size \alpha and a small \epsilon for stability:

\theta_t = \theta_{t-1} - \alpha \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}

The paper recommends \alpha = 0.001, \beta_1 = 0.9, \beta_2 = 0.999, and \epsilon = 10^{-8} — defaults that transfer remarkably well across models and tasks.

Step t gradient g 1st moment (β₁) m — momentum 2nd moment (β₂) v — scaling Bias correction m̂ / (√v̂ + ε) Update θ − α·…
Adam: each gradient updates a momentum estimate and a per-parameter scale estimate; bias-corrected versions combine into the final update step.

Why per-parameter rates help

Different parameters in a deep network live on wildly different scales. Embedding rows for rare words may receive gradients once in a thousand steps, while top-layer weights update on every batch; a single global learning rate is either too timid for the former or too aggressive for the latter. By dividing each parameter's step by the root-mean-square of its own gradient history, Adam automatically takes larger effective steps for sparse, quiet parameters and smaller ones for noisy, active parameters. This is why Adam shines on problems with sparse gradients — natural language models, embeddings, and attention layers — where plain stochastic gradient descent needs extensive per-layer tuning to keep up. The update magnitudes are also approximately bounded by the step size \alpha, which makes training behavior predictable even when gradient scales shift across layers.

What the paper showed

Kingma and Ba tested Adam on logistic regression, multilayer perceptrons, and convolutional networks, comparing against SGD with momentum, AdaGrad, RMSProp, and AdaDelta. Adam converged faster and reached comparable or better final loss across these tasks, and it proved robust to the choice of hyperparameters — the defaults worked nearly everywhere. The authors also analyzed the algorithm theoretically, showing a regret bound comparable to the best known online convex optimization results. Just as important was a practical property: Adam needs only first-order gradients and modest memory (two extra numbers per parameter), so it drops into any existing training loop at negligible cost. Within a few years it had become the optimizer researchers reached for first, and large-scale projects from ResNet-era vision models to Transformer pretraining adopted it or its descendants.

AdamW and the weight-decay fix

One wart emerged with regularization. The standard way to add L2 regularization is to fold the weight-decay gradient into g_t, but in Adam that decay term gets rescaled by the adaptive denominator — parameters with large gradient histories receive less regularization than quiet ones, which is rarely what anyone wants. Loshchilov and Hutter's 2017 fix, AdamW, decouples the two: the adaptive update proceeds as before, and weight decay is applied directly to the parameters afterward. This small change improved generalization noticeably, and AdamW has since become the standard optimizer for training Transformers and most large models. Modern libraries implement AdamW as the default Adam variant, so when a paper says "we trained with Adam," it very often means AdamW in practice.

Limitations and practical advice

Adam is not universally best. On some image-classification benchmarks, carefully tuned SGD with momentum still reaches slightly better final accuracy than Adam, possibly because adaptive methods can converge to sharper minima that generalize worse. Adam can also behave badly very early in training, when the variance estimate v_t is built from few samples — learning-rate warmup, where \alpha ramps up over the first few thousand steps, is the standard remedy and is near-universal in large-model training. Practical guidance: start with AdamW, the default betas, and a short warmup; tune the peak learning rate first and the decay schedule second; and consider switching to SGD with momentum for a final comparison on vision tasks where every tenth of a point matters.

Source

  • Kingma & Ba, Adam: A Method for Stochastic Optimization, 2014. arXiv:1412.6980
  • Loshchilov & Hutter, Decoupled Weight Decay Regularization, 2017. arXiv:1711.05101
  • Ioffe & Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, 2015. arXiv:1502.03167

Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.

Cite this report
@misc{knowyourml-adam-adaptive-moment-estimation
  title        = {Adam: Adaptive Moment Estimation},
  author       = {{Know Your ML}},
  year         = {2014},
  howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=adam-adaptive-moment-estimation}},
  note         = {AI-generated review of arXiv:1412.6980. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit — decision matrix, scorecards and trade-off analysis
Toolkit 7 decision tools Model-agnostic

Machine Learning Decision Toolkit

Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.

$24 View on Gumroad

References

Frequently asked questions

How does Adam combine momentum and RMSProp?

Momentum accelerates along consistent gradient directions by averaging past gradients, while RMSProp-style scaling divides each parameter's update by the size of its recent gradients. Adam keeps both: a moving average of the gradients and a moving average of the squared gradients.

Why does Adam need bias correction?

Both averages start at zero, so early estimates are biased toward zero. Adam divides each average by a correction factor that starts small and grows to one, giving unbiased estimates from the very first step.

What are Adam's default hyperparameters?

The paper suggests alpha = 0.001, beta1 = 0.9, beta2 = 0.999, and epsilon = 1e-8. These defaults work across a wide range of models, which is a large part of why Adam is so popular.

What is the difference between Adam and AdamW?

AdamW decouples weight decay from the adaptive update so regularization applies evenly to all parameters instead of being rescaled by the gradient history. It has largely replaced Adam with L2 regularization for training Transformers.

How to Remain Valuable When Intelligence Becomes Cheap — book cover
eBook 224 pages

How to Remain Valuable When Intelligence Becomes Cheap

This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?

Read the eBook