AI engineering glossary · Evaluation metrics
What is relevance judgments?
Relevance judgments (also called qrels): Human labels stating which documents are relevant to each test query, sometimes on a graded scale. Together with the queries and the document collection they form the ground truth that ranking metrics such as MAP and NDCG are computed against.
Relevance judgments: an example
An Indian legal-search test set has 50 queries, each with 20 judged court cases marked 0 (irrelevant) to 3 (highly relevant).
Related terms
- Cranfield paradigm
- Normalised discounted cumulative gain
- Mean average precision
- Inter-annotator agreement
More Evaluation metrics terms
- Reliability diagram
- Residual plot
- ROC curve
- Root mean squared error
- Root mean squared logarithmic error
- ROUGE
- ROUGE-L
- Slice-based evaluation
Where you learn it
Relevance judgments is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.