AI engineering glossary · Statistics inference
What is multiple comparisons problem?
Multiple comparisons problem: The rise in false positives when many hypotheses are tested at once: each test has its own small chance of a false alarm, and these accumulate. Without adjustment, some significant results are expected purely by luck.
Multiple comparisons problem: an example
Testing 20 unrelated metrics at the 5 percent level gives a 1 - 0.95^20, about 64 percent, chance of at least one false alarm.
Related terms
More Statistics inference terms
- Non-parametric test
- Novelty effect
- Null hypothesis
- One-tailed and two-tailed tests
- P-hacking
- P-value
- Paired t-test
- Peeking problem
Where you learn it
Multiple comparisons problem is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.