AI engineering glossary · Evals guardrails AI security
What is online evaluation?
Online evaluation (also called production evaluation): Measuring a system's quality on live user traffic, using signals such as user feedback, task completion, sampled human review or automated judges scoring production outputs. It reveals problems that offline sets miss, such as new phrasing and unexpected user behaviour.
Online evaluation: an example
An e-commerce assistant samples 2 percent of conversations each day and has a judge model score them for helpfulness and policy violations.
Related terms
More Evals guardrails AI security terms
- Output guardrail
- Over-refusal
- Overreliance
- OWASP Top 10 for LLM Applications
- Pairwise comparison
- Payload splitting
- Perplexity filter
- Pickle deserialization risk
Where you learn it
Online evaluation is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.