AI engineering glossary · Evaluation metrics
What is fleiss' kappa?
Fleiss' kappa: An extension of Cohen's kappa that measures chance-corrected agreement among a fixed number of raters (more than two) assigning categories to items. Each item need not be rated by the same people, but each must have the same number of ratings. 1 is perfect agreement and 0 is chance level.
Fleiss' kappa: an example
Five annotators each label 300 reviews as positive, neutral or negative; Fleiss' kappa summarises how far their agreement exceeds chance.
Related terms
More Evaluation metrics terms
- G-mean
- Gini coefficient
- Hamming loss
- Hit rate
- Human evaluation
- Isotonic calibration
- Jaccard index
- Kendall's tau
Where you learn it
Fleiss' kappa is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.