AI engineering glossary · Evals guardrails AI security
What is lm-evaluation-harness?
lm-evaluation-harness: An open-source framework from EleutherAI for evaluating language models on many standard benchmark tasks with a common interface. It is widely used to reproduce published results and to compare open models under identical prompts and settings, since small formatting differences can change scores.
lm-evaluation-harness: an example
A researcher runs the harness on a fine-tuned 7B model against several benchmark tasks and compares the table with the base model.
Related terms
More Evals guardrails AI security terms
- Many-shot jailbreaking
- Membership inference attack
- Misinformation risk
- MITRE ATLAS
- MMLU
- Model extraction
- Model inversion
- Model signing
Where you learn it
lm-evaluation-harness is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.