Automated LLM evaluation pipelines: from error analysis to CI

Short answer: an LLM evaluation pipeline runs your AI feature on a fixed set of test inputs every time something changes (prompt, model, retrieval, code) and scores the outputs automatically, so a regression is caught before users see it. Build it in five steps: look at real failures (error analysis), turn them into a golden dataset, write code-based checks where you can, use an LLM judge calibrated against human labels where you cannot, and run it all in CI with a pass threshold.

Step 1: error analysis first

Read 50 to 100 real traces (inputs, retrieved context, outputs) and write down what went wrong in plain words. Group the notes into failure types: wrong facts, missing citations, wrong format, refused a valid request, too long. These categories become your metrics. Metrics chosen before looking at data usually measure the wrong thing.

Step 2: a golden dataset

For each failure type, collect examples with the expected behaviour: the question, the context and what a good answer must contain or avoid. Mix real cases with synthetic ones for rare situations. Version the dataset like code.

Step 3: code-based checks (cheap, exact)

• Format: valid JSON, required fields, length limits.
• Content: must mention X, must not mention Y, the cited source ID exists in the retrieved set.
• Retrieval: recall@k and MRR for the retriever, measured separately from the answer.

def check(answer, case):
    assert answer["refund_days"] == case["expected_days"]
    assert all(c in case["retrieved_ids"] for c in answer["citations"])

Step 4: LLM-as-a-judge (for what code cannot check)

Faithfulness, helpfulness and tone need judgement. Use a judge prompt with a clear rubric and a structured verdict, then calibrate it: label 50 to 100 outputs by hand, compare with the judge, and fix the rubric until they agree most of the time. Use pass/fail per criterion rather than a vague 1-10 score. A runnable judge skeleton is on the LLM-as-a-judge glossary page.

Step 5: evals in CI

Run the suite on every pull request that touches prompts, models or retrieval. Fail the build when the pass rate drops below the threshold or any critical case fails. Store each run's scores so you can see trends, and add every new production failure to the dataset (that is how the suite grows teeth).

Tracing and monitoring

Offline evals catch known failures; production needs tracing (every model call, tool call and retrieval logged with inputs, outputs, latency and cost) and sampling real traffic back into error analysis. Open-source tracing tools make this straightforward.

Where this is taught

Module 9 (Evals, Guardrails & AI Security): error analysis on real traces, golden and synthetic test data, code-based checks, LLM-as-judge calibrated against human labels, evals in CI, tracing and observability. Its project is an evaluation suite and red-team report wired into CI. Related: How to write guardrails · LLM evaluation and guardrails course.