AI engineering glossary · Evals guardrails AI security
What is red teaming?
Red teaming (also called AI red teaming): Deliberately attacking an AI system, as an adversary would, to uncover harmful outputs, security holes and misuse paths before real users or attackers find them. Testers try jailbreaks, injections and abuse scenarios, and findings feed fixes, guardrails and new test cases.
Red teaming: an example
A fintech's red team spends a week trying to make its loan chatbot leak customer data or give unlawful advice, logging each success.
Related terms
More Evals guardrails AI security terms
- Reference-based evaluation
- Reference-free evaluation
- Refusal rate
- Role-play jailbreak
- Rubric-based grading
- Safety evaluation
- Sandboxing
- Schema validation
Where you learn it
Red teaming is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.