Overfitting vs Underfitting: How to Tell Which One Your Model Has
Underfitting is a model too simple to learn the pattern. Overfitting is a model that has learned its training examples, noise included. Comparing training error with validation error tells you which one you have.
Underfitting happens when a model is too simple, too constrained or stopped too early to capture the pattern in its data. It is then wrong on the examples it trained on and on new ones. Overfitting happens when a model has fitted the particular training examples, including their random noise. Its error on the training data looks excellent, but on new data it is much worse. The two failures look different on paper, and each has a different fix.
The quickest test is to compare the error on the training data with the error on a validation set, which the model never trains on. If both errors are high and close together, the model underfits. If training error is low and validation error is much higher, the model overfits.
Two ways a model can miss
Goodfellow, Bengio and Courville frame the problem as two goals: make the training error small, and make the gap between training and test error small. Those two goals correspond to the two central challenges of machine learning, underfitting and overfitting (Goodfellow et al., 2016, chapter 5). Underfitting is failing the first goal. Overfitting is failing the second.
An underfitting model cannot get its training error low. It lacks capacity, is missing the information it needs, or was stopped before it learned enough. Think of a student who never learned the material: they do badly on practice questions and on the exam.
An overfitting model gets its training error low, but the gap to new data is large. It has learned details of the training set that do not repeat anywhere else. That is a student who memorised last year's answer key: flawless on the old questions, lost when the questions change.
See it with a polynomial fit
Polynomial regression shows both failures in one picture. A polynomial of degree 1 is a straight line, degree 2 is a parabola, and each higher degree can bend more often. Fit noisy points from a smooth curve with a low degree and the line cannot follow the bend. Fit with a very high degree and the curve passes close to every point while swinging between them.
scikit-learn's underfitting-versus-overfitting example uses the same setup in code. A linear function (degree 1) is not sufficient to fit the training samples, which is underfitting. A polynomial of degree 4 approximates the true function almost perfectly. Higher degrees overfit: they learn the noise in the training data. The example then scores each degree with cross-validation rather than trusting the training fit (scikit-learn documentation).
The playground's Polynomial Fit (Overfitting) tool does this interactively. Add points or generate random ones, then move the degree slider from 1 to 12, and watch which parts of the curve follow the data and which invent shape between the points.
Read the learning curve
Training error on its own cannot diagnose a model, because it tends to fall as capacity grows. You need held-out error next to it. Hold back a validation set, then record both errors as you change the model size, the training time or the amount of data. Plotting the two against model complexity gives a learning curve, and its shape tells you what to do next.
| What you see | Diagnosis | Usual first move |
|---|---|---|
| Training and validation error both high, and close together | Underfitting | More capacity or better inputs, longer training, weaker regularisation |
| Training and validation error both low, and close together | A good fit, for now | Check the test set once, then stop tuning |
| Training error low, validation error much higher | Overfitting | More varied data, regularisation, early stopping, or a smaller model |
| Training error still falling, validation error rising | Overfitting has started | Stop at the validation minimum and keep that checkpoint |
Why a big model is not automatically overfitting
Model size is a poor stand-in for overfitting. Zhang and colleagues showed that standard image-classification networks, trained with stochastic gradient methods, can fit a random labelling of their training images. Their capacity to memorise is therefore very large. They also argued that the usual explanations, the model family and explicit regularisation, do not by themselves account for why large networks generalise well in practice (Zhang et al., 2017).
The practical lesson is that overfitting has to be measured on held-out data. Counting parameters tells you little about whether a particular trained model will generalise.
Fixing underfitting
- Add capacity or information. Use more layers or units, or inputs that expose the pattern. A model cannot use signal it never receives.
- Train longer, but check the curve. If training error is still falling, more epochs may help. If it is flat and high, more training will not.
- Weaken regularisation. A large weight penalty or a high dropout rate caps what the model can learn.
- Check the data before the model. If both errors stay high after capacity increases, the labels may be noisy, the inputs may lack the signal, or the task may simply be harder than it looks.
Fixing overfitting
- More varied data. A larger, representative training set narrows the gap. It helps less when the examples are biased or mislabelled.
- Dropout. Srivastava et al. (2014) randomly drop units, along with their connections, from the network during training. The Dropout report on this site covers the same idea.
- Mixup. Zhang, Cisse, Dauphin and Lopez-Paz (2017) train on convex combinations of pairs of examples and their labels, which pushes the model toward simple linear behaviour between training examples. See the mixup report.
- Early stopping. Goodfellow et al. describe continuing training while training error falls, then stopping once validation error has not improved for a set number of iterations. They call it probably the most commonly used form of regularisation in deep learning.
- A smaller or more constrained model, once a larger one is clearly memorising its training set.
A checklist you can run
- Split first. Set aside a validation set and a test set before you look at any result, and use the test set once, at the end. If your data has a time order, split by time, or the validation score will be optimistic.
- Log both errors. Record training and validation error for every model size or training step you compare.
- Read the gap, then change one thing. Apply a single fix and measure again. Several changes at once make it impossible to tell which one helped.
- Stop at the validation minimum. Keep the checkpoint with the lowest validation error, not the one with the lowest training error.
Where to go next
The playground is the fastest way to build intuition for fit. For the training side, the training topic hub lists its six reports, including the dropout, mixup and normalisation work mentioned above. The Scaling Laws report shows how language-model loss follows power laws in parameters, data and compute. For everything else, start at the reports directory, or read how this site checks its sources.
References
- Goodfellow, I., Bengio, Y. and Courville, A. (2016). Deep Learning. MIT Press. Chapter 5, machine learning basics: https://www.deeplearningbook.org/contents/ml.html. Chapter 7, regularisation (early stopping): https://www.deeplearningbook.org/contents/regularization.html.
- Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. and Salakhutdinov, R. (2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research 15: https://jmlr.org/papers/v15/srivastava14a.html.
- Zhang, C., Bengio, S., Hardt, M., Recht, B. and Vinyals, O. (2017). Understanding deep learning requires rethinking generalization. arXiv:1611.03530: https://arxiv.org/abs/1611.03530.
- Zhang, H., Cisse, M., Dauphin, Y. N. and Lopez-Paz, D. (2017). mixup: Beyond Empirical Risk Minimization. arXiv:1710.09412: https://arxiv.org/abs/1710.09412.
- scikit-learn developers. Underfitting vs. Overfitting (example gallery): https://scikit-learn.org/stable/auto_examples/model_selection/plot_underfitting_overfitting.html.
This guide is an explainer, not peer-reviewed research. Check the linked primary sources before relying on any specific claim.
Frequently asked questions
What is the simplest way to tell overfitting from underfitting?
Compare the error on training data with the error on a validation set the model never trained on. If both are high and close together, the model underfits. If training error is much lower than validation error, the model overfits.
Is a low training error a sign of a good model?
Only if the validation error is low too. A model can memorise its training set and reach very low training error while generalising poorly. Zhang et al. (2017) showed that standard image networks can fit random labels of their training images, so fitting the training set is not evidence of generalisation on its own.
Does more data always fix overfitting?
More representative data usually narrows the gap between training and validation error, but it is not a guaranteed fix. Biased or mislabelled examples, and a mismatch between the training data and the data the model meets later, remain problems however large the dataset.
How is dropout different from early stopping?
Dropout changes the training computation by randomly removing units from the network during each update. Early stopping ends training when validation error stops improving. Both are regularisers, and they are often used together.
Why does validation error rise after it has fallen?
That is the usual sign that the model has started fitting details of the training set that do not carry over to new data. Keep the checkpoint with the lowest validation error, add regularisation, or use a smaller model. Also check that the validation set really resembles the data you expect to see later.