AI engineering glossary · ML fundamentals
What is data snooping?
Data snooping (also called Adaptive overfitting): Repeatedly using the same held-out data to make modelling choices, so that the reported score reflects choices tuned to that data rather than true performance. Each look at the test results makes the estimate a little less trustworthy.
Data snooping: an example
After trying 40 model variants and always picking the one with the best test score, the winning score is inflated by selection.
Related terms
More ML fundamentals terms
- Data-centric AI
- Dataset
- Decision threshold
- Discriminative model
- Distribution shift
- Double descent
- Early stopping
- Empirical risk minimisation
Where you learn it
Data snooping is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.