AI engineering glossary · Evals guardrails AI security
What is humanEval?
HumanEval: A code-generation benchmark of hand-written Python programming problems, each with unit tests, used to check whether a model writes functionally correct code. Results are commonly reported as pass@k, the chance that at least one of k sampled solutions passes all tests.
HumanEval: an example
A model writes a function to reverse words in a sentence; it counts as solved only if the hidden unit tests all pass.
Related terms
More Evals guardrails AI security terms
- Improper output handling
- Input guardrail
- Intrinsic hallucination
- ISO/IEC 42001
- Jailbreak
- Judge calibration
- Least privilege
- Llama Guard
Where you learn it
HumanEval is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.