AI engineering glossary · Evaluation metrics
What is krippendorff's alpha?
Krippendorff's alpha: A flexible agreement measure that works with any number of raters, missing ratings and different types of scale (nominal, ordinal, interval or ratio). It is defined as 1 minus the ratio of observed to expected disagreement. It is often preferred for real annotation projects with incomplete data.
Krippendorff's alpha: an example
In a project where each of 1,000 sentences is rated by only 3 of 8 annotators, alpha can still be computed, while simpler kappas cannot.
Related terms
More Evaluation metrics terms
- Likelihood ratio
- Log loss
- Macro averaging
- Matthews correlation coefficient
- MAUVE
- Max error
- Maximum calibration error
- McNemar's test
Where you learn it
Krippendorff's alpha is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.