AI engineering glossary · ML fundamentals
What is overfitting?
Overfitting: When a model learns the quirks and random noise of its training data so closely that it performs worse on new data. It shows up as very low training error but much higher error on held-out data. Causes include overly flexible models, too little data and too many features.
Overfitting: an example
A decision tree grown until every training row has its own leaf scores 100% on training data but only 70% on new customers.
Overfitting in Python
A short example you can run (Python 3.11, scikit-learn). The output below is what it really prints.
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
X, y = make_classification(n_samples=400, n_features=20, flip_y=0.2, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, random_state=0)
for depth in (2, None):
m = DecisionTreeClassifier(max_depth=depth, random_state=0).fit(Xtr, ytr)
print(f"max_depth={depth}: train {m.score(Xtr, ytr):.2f}, test {m.score(Xte, yte):.2f}")Output
max_depth=2: train 0.80, test 0.76 max_depth=None: train 1.00, test 0.69
Related terms
More ML fundamentals terms
- Overparameterisation
- PAC learning
- Parameter
- Parametric model
- Pipeline
- Probabilistic model
- Problem framing
- Random search
Where you learn it
Overfitting is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.