AI engineering glossary · Evaluation metrics
What is pass@k?
pass@k (also called pass at k): A metric for code generation: the probability that at least one of k sampled programs passes all unit tests for a problem. It is usually estimated from n samples with c correct as 1 - C(n-c, k) / C(n, k), then averaged across problems.
pass@k: an example
If 3 of 10 generated solutions pass, pass@1 is 0.3, and pass@5 is higher because any of five attempts may succeed.
Related terms
More Evaluation metrics terms
- Perplexity
- Pinball loss
- Platt scaling
- Precision at k
- Precision-recall break-even point
- Precision-recall curve
- Precision-recall trade-off
- Prediction interval coverage
Where you learn it
pass@k is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.