AI engineering glossary · Evals guardrails AI security
What is pairwise comparison?
Pairwise comparison (also called A/B comparison, head-to-head evaluation): An evaluation format where a grader, human or model, sees two candidate answers to the same prompt and chooses the better one or declares a tie. Relative choices are easier and more consistent than absolute scores, and the win rates they produce can be turned into rankings between models or prompt versions.
Pairwise comparison: an example
Shown answers A and B to the same loan-FAQ question, a judge picks B; across 200 questions, prompt version 2 wins 128 times, a 64 percent win rate.
Related terms
More Evals guardrails AI security terms
- Payload splitting
- Perplexity filter
- Pickle deserialization risk
- PII detection
- PII redaction
- Prompt injection
- Prompt leaking
- promptfoo
Where you learn it
Pairwise comparison is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.