AI engineering glossary · Evals guardrails AI security
What is SWE-bench?
SWE-bench: A benchmark built from real GitHub issues in popular Python repositories, where a system must edit the codebase to resolve an issue and is scored by whether the project's tests then pass. It tests realistic software-engineering ability, including reading large codebases, rather than isolated function writing.
SWE-bench: an example
An agent receives a bug report from an open-source library, produces a code patch, and is counted as successful if the relevant tests pass.
Related terms
More Evals guardrails AI security terms
- Synthetic test data
- System prompt extraction
- Threat modelling
- Tool poisoning
- Tool-call accuracy
- Topical rails
- Toxicity classification
- Training data extraction
Where you learn it
SWE-bench is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.