AI engineering glossary · ML fundamentals
What is inter-annotator agreement?
Inter-annotator agreement (also called Inter-rater reliability): A measure of how consistently different people give the same label to the same items, often summarised with a statistic such as Cohen's kappa. Low agreement suggests unclear guidelines or an inherently subjective task, and puts a ceiling on model quality.
Inter-annotator agreement: an example
Two annotators agree on 88 of 100 review labels; kappa adjusts that figure for agreement expected by chance.
Related terms
More ML fundamentals terms
- Irreducible error
- k-fold cross-validation
- L1 regularisation
- L2 regularisation
- Label
- Label noise
- Latent variable
- Learning curve
Where you learn it
Inter-annotator agreement is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.