Explainer for ML learners 8 min read

Overfitting vs Underfitting: How to Tell Which One Your Model Has

Underfitting is a model too simple to learn the pattern. Overfitting is a model that has learned its training examples, noise included. Comparing training error with validation error tells you which one you have.

overfitting underfitting validation error regularisation
Black-and-white photograph of a tailor adjusting the collar of a uniform coat on a man wearing a peaked cap, in a workshop with pipes on the wall.
A tailor checks how one coat fits one person. A model is fitted to data in the same way: the aim is a fit that holds for the next person, not only the one it was measured on. Photograph by Esther Bubley, September 1943, Library of Congress (FSA/OWI collection). Public domain. Source: Wikimedia Commons.

Underfitting happens when a model is too simple, too constrained or stopped too early to capture the pattern in its data. It is then wrong on the examples it trained on and on new ones. Overfitting happens when a model has fitted the particular training examples, including their random noise. Its error on the training data looks excellent, but on new data it is much worse. The two failures look different on paper, and each has a different fix.

The quickest test is to compare the error on the training data with the error on a validation set, which the model never trains on. If both errors are high and close together, the model underfits. If training error is low and validation error is much higher, the model overfits.

Two ways a model can miss

Goodfellow, Bengio and Courville frame the problem as two goals: make the training error small, and make the gap between training and test error small. Those two goals correspond to the two central challenges of machine learning, underfitting and overfitting (Goodfellow et al., 2016, chapter 5). Underfitting is failing the first goal. Overfitting is failing the second.

An underfitting model cannot get its training error low. It lacks capacity, is missing the information it needs, or was stopped before it learned enough. Think of a student who never learned the material: they do badly on practice questions and on the exam.

An overfitting model gets its training error low, but the gap to new data is large. It has learned details of the training set that do not repeat anywhere else. That is a student who memorised last year's answer key: flawless on the old questions, lost when the questions change.

Illustration of the three bears at their table, with bowls and chairs of different sizes.
Arthur Rackham's illustration of the Goldilocks story. Goldilocks has to try the alternatives before she finds the one that fits. Choosing a model works the same way: compare a too-simple model, a too-flexible one and an in-between one on data they were not trained on. Illustration by Arthur Rackham, public domain. Source: Wikimedia Commons (Project Gutenberg eText 17034).

See it with a polynomial fit

Polynomial regression shows both failures in one picture. A polynomial of degree 1 is a straight line, degree 2 is a parabola, and each higher degree can bend more often. Fit noisy points from a smooth curve with a low degree and the line cannot follow the bend. Fit with a very high degree and the curve passes close to every point while swinging between them.

scikit-learn's underfitting-versus-overfitting example uses the same setup in code. A linear function (degree 1) is not sufficient to fit the training samples, which is underfitting. A polynomial of degree 4 approximates the true function almost perfectly. Higher degrees overfit: they learn the noise in the training data. The example then scores each degree with cross-validation rather than trusting the training fit (scikit-learn documentation).

The playground's Polynomial Fit (Overfitting) tool does this interactively. Add points or generate random ones, then move the degree slider from 1 to 12, and watch which parts of the curve follow the data and which invent shape between the points.

Degree 1 fit: a straight line that misses the curve the points came from Degree 1 too simple: underfits Degree 4 fit: a curve that follows the dashed line closely Degree 4 close to the curve Degree 10 fit: a curve that passes near every point and swings away between them Degree 10 chases the noise
Illustrative, not measured. Twelve synthetic points were drawn from a sine curve with added noise and fitted by least squares with polynomials of degree 1, 4 and 10. Dashed grey: the curve the points came from. Dots: the points. Rose: the fitted polynomial. Degree 1 misses the shape, degree 4 follows it, and degree 10 passes close to the points while swinging away between them and at the edges.

Read the learning curve

Training error on its own cannot diagnose a model, because it tends to fall as capacity grows. You need held-out error next to it. Hold back a validation set, then record both errors as you change the model size, the training time or the amount of data. Plotting the two against model complexity gives a learning curve, and its shape tells you what to do next.

How to read training and validation error together
What you see Diagnosis Usual first move
Training and validation error both high, and close together Underfitting More capacity or better inputs, longer training, weaker regularisation
Training and validation error both low, and close together A good fit, for now Check the test set once, then stop tuning
Training error low, validation error much higher Overfitting More varied data, regularisation, early stopping, or a smaller model
Training error still falling, validation error rising Overfitting has started Stop at the validation minimum and keep that checkpoint
Schematic learning curves for training and validation error Training error falls steadily as model complexity grows. Validation error falls, reaches a lowest point at moderate complexity, then rises again. The gap between the two curves on the right is the overfitting signal. error model complexity or training time training error validation error both high: underfitting gap: overfitting
Schematic, not measured. The typical shape when error is plotted against model complexity or training time: training error keeps falling, while validation error bottoms out and then rises. The space between the two curves on the right is the overfitting signal.

Why a big model is not automatically overfitting

Model size is a poor stand-in for overfitting. Zhang and colleagues showed that standard image-classification networks, trained with stochastic gradient methods, can fit a random labelling of their training images. Their capacity to memorise is therefore very large. They also argued that the usual explanations, the model family and explicit regularisation, do not by themselves account for why large networks generalise well in practice (Zhang et al., 2017).

The practical lesson is that overfitting has to be measured on held-out data. Counting parameters tells you little about whether a particular trained model will generalise.

Fixing underfitting

  • Add capacity or information. Use more layers or units, or inputs that expose the pattern. A model cannot use signal it never receives.
  • Train longer, but check the curve. If training error is still falling, more epochs may help. If it is flat and high, more training will not.
  • Weaken regularisation. A large weight penalty or a high dropout rate caps what the model can learn.
  • Check the data before the model. If both errors stay high after capacity increases, the labels may be noisy, the inputs may lack the signal, or the task may simply be harder than it looks.

Fixing overfitting

  • More varied data. A larger, representative training set narrows the gap. It helps less when the examples are biased or mislabelled.
  • Dropout. Srivastava et al. (2014) randomly drop units, along with their connections, from the network during training. The Dropout report on this site covers the same idea.
  • Mixup. Zhang, Cisse, Dauphin and Lopez-Paz (2017) train on convex combinations of pairs of examples and their labels, which pushes the model toward simple linear behaviour between training examples. See the mixup report.
  • Early stopping. Goodfellow et al. describe continuing training while training error falls, then stopping once validation error has not improved for a set number of iterations. They call it probably the most commonly used form of regularisation in deep learning.
  • A smaller or more constrained model, once a larger one is clearly memorising its training set.

A checklist you can run

  1. Split first. Set aside a validation set and a test set before you look at any result, and use the test set once, at the end. If your data has a time order, split by time, or the validation score will be optimistic.
  2. Log both errors. Record training and validation error for every model size or training step you compare.
  3. Read the gap, then change one thing. Apply a single fix and measure again. Several changes at once make it impossible to tell which one helped.
  4. Stop at the validation minimum. Keep the checkpoint with the lowest validation error, not the one with the lowest training error.

Where to go next

The playground is the fastest way to build intuition for fit. For the training side, the training topic hub lists its six reports, including the dropout, mixup and normalisation work mentioned above. The Scaling Laws report shows how language-model loss follows power laws in parameters, data and compute. For everything else, start at the reports directory, or read how this site checks its sources.

References

This guide is an explainer, not peer-reviewed research. Check the linked primary sources before relying on any specific claim.

Frequently asked questions

What is the simplest way to tell overfitting from underfitting?

Compare the error on training data with the error on a validation set the model never trained on. If both are high and close together, the model underfits. If training error is much lower than validation error, the model overfits.

Is a low training error a sign of a good model?

Only if the validation error is low too. A model can memorise its training set and reach very low training error while generalising poorly. Zhang et al. (2017) showed that standard image networks can fit random labels of their training images, so fitting the training set is not evidence of generalisation on its own.

Does more data always fix overfitting?

More representative data usually narrows the gap between training and validation error, but it is not a guaranteed fix. Biased or mislabelled examples, and a mismatch between the training data and the data the model meets later, remain problems however large the dataset.

How is dropout different from early stopping?

Dropout changes the training computation by randomly removing units from the network during each update. Early stopping ends training when validation error stops improving. Both are regularisers, and they are often used together.

Why does validation error rise after it has fallen?

That is the usual sign that the model has started fitting details of the training set that do not carry over to new data. Keep the checkpoint with the lowest validation error, add regularisation, or use a smaller model. Also check that the validation set really resembles the data you expect to see later.