AI engineering glossary · Evaluation metrics
What is cranfield paradigm?
Cranfield paradigm (also called test collection methodology): The classic method for evaluating search systems using a fixed test collection: a set of documents, a set of queries and human relevance judgments. Any system can be re-run against it and scored repeatably, which makes results comparable across systems and over time.
Cranfield paradigm: an example
To compare two search engines, run the same 100 queries on both against one document set and judgments file, then compare their MAP scores.
Related terms
More Evaluation metrics terms
- Cumulative gains chart
- Davies-Bouldin index
- Diagnostic odds ratio
- Discounted cumulative gain
- Dunn index
- Error analysis
- Exact match
- Expected calibration error
Where you learn it
Cranfield paradigm is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.