AI engineering glossary · Statistics inference
What is peeking problem?
Peeking problem (also called Optional stopping): The inflation of false positives that happens when an experiment's p-value is checked repeatedly and stopped as soon as it looks significant. A fixed-sample test is valid only for one look at the planned sample size.
Peeking problem: an example
Checking a test daily and stopping the first time p < 0.05 can push the true false-positive rate far above 5 percent.
Related terms
More Statistics inference terms
- Permutation test
- Point estimate
- Population parameter
- Post-hoc test
- Posterior distribution
- Potential outcomes framework
- Power analysis
- Practical significance
Where you learn it
Peeking problem is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.