AI engineering glossary · Evals guardrails AI security
What is adversarial test set?
Adversarial test set: A collection of inputs deliberately designed to make a system fail, such as tricky phrasing, injection attempts, harmful requests or ambiguous cases. Scoring it shows weaknesses that a typical eval set hides, and it is used to track safety progress across releases.
Adversarial test set: an example
A bank adds 100 prompts that try to trick its assistant into revealing another customer's balance and tracks how many succeed each release.
Related terms
More Evals guardrails AI security terms
- Adversarial training
- Answer relevancy
- Attack success rate
- Attack surface
- Automated red teaming
- Backdoor attack
- Benchmark saturation
- Canary token
Where you learn it
Adversarial test set is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.