AI engineering glossary · Evaluation metrics
What is reliability diagram?
Reliability diagram (also called calibration curve): A plot for checking calibration: predictions are grouped into confidence bins, and each bin's average predicted probability is plotted against the actual fraction of positives. Points on the diagonal mean perfect calibration, below it means overconfident, above it underconfident.
Reliability diagram: an example
For the bin of predictions near 0.8, only 60% turn out positive, so the point falls well below the diagonal, showing overconfidence.
Related terms
More Evaluation metrics terms
- Residual plot
- ROC curve
- Root mean squared error
- Root mean squared logarithmic error
- ROUGE
- ROUGE-L
- Slice-based evaluation
- Spearman's rank correlation
Where you learn it
Reliability diagram is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.