AI engineering glossary · Evaluation metrics
What is human evaluation?
Human evaluation (also called human eval, human judgment): Having people rate or compare model outputs on criteria like fluency, accuracy or helpfulness, often on a Likert scale or through pairwise preference. It is slow and costly, but remains the reference standard against which automatic metrics are validated.
Human evaluation: an example
Three annotators rate 200 chatbot answers from 1 to 5 for helpfulness, and their average is reported.
Related terms
More Evaluation metrics terms
- Isotonic calibration
- Jaccard index
- Kendall's tau
- Kolmogorov-Smirnov statistic
- Krippendorff's alpha
- Likelihood ratio
- Log loss
- Macro averaging
Where you learn it
Human evaluation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.