AI engineering glossary · Evaluation metrics
What is ECE (expected calibration error)?
Expected calibration error (also called ECE): A single number summarising miscalibration: predictions are split into bins, and the gap between each bin's average confidence and accuracy is averaged, weighted by bin size: ECE = sum of (n_b / N) * |accuracy_b - confidence_b|. Lower is better, and 0 is perfect.
Expected calibration error: an example
If a model is 90% confident on a bin holding half the data but only 80% accurate there, that bin contributes 0.5 * 0.10 = 0.05 to ECE.
Related terms
More Evaluation metrics terms
- Expected reciprocal rank
- Explained variance score
- F-beta score
- F1 score
- False negative
- False negative rate
- False positive
- False positive rate
Where you learn it
Expected calibration error is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.