What is LLM-as-a-judge? Meaning and example in Python

What is LLM-as-a-judge?

LLM-as-a-judge (also called LLM judge, model-graded evaluation): A technique where one language model is prompted to grade the outputs of another model or system, scoring them against a rubric or picking the better of two answers. It is far cheaper and faster than human review and scales to thousands of cases, but the judge has its own biases and must be checked against human labels.

LLM-as-a-judge: an example

A judge model reads 500 chatbot answers and gives each a 1 to 5 score for helpfulness; the team compares 50 of those scores with human ratings to confirm agreement.

LLM-as-a-judge in Python

A short example you can run (Python 3.11). The output below is what it really prints.

JUDGE_PROMPT = """Score the answer from 1 to 5 for faithfulness to the context.
Context: {context}
Answer: {answer}
Reply as JSON: {{"score": <1-5>, "reason": "<one sentence>"}}"""

import json
def judge(context, answer):
    prompt = JUDGE_PROMPT.format(context=context, answer=answer)
    # reply = llm(prompt)  <- a real judge model; here a fixed reply so the code runs offline
    reply = '{"score": 2, "reason": "The answer says 30 days but the context says 7."}'
    return json.loads(reply)

print(judge("Refunds within 7 days.", "You can get a refund within 30 days."))

Output

{'score': 2, 'reason': 'The answer says 30 days but the context says 7.'}

Related terms

More Evals guardrails AI security terms

Where you learn it

LLM-as-a-judge is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.