AI engineering glossary · Evals guardrails AI security
What is MMLU?
MMLU: Massive Multitask Language Understanding, a benchmark of multiple-choice questions spanning dozens of academic and professional subjects, such as law, medicine and mathematics. It measures broad knowledge and reasoning and has been a standard headline number for language models, though top models have largely saturated it.
MMLU: an example
A model answers 14,000 multiple-choice questions and is reported as getting a given percentage right, averaged over subjects.
Related terms
More Evals guardrails AI security terms
- Model extraction
- Model inversion
- Model signing
- Model supply chain attack
- Multimodal prompt injection
- NeMo Guardrails
- NIST AI Risk Management Framework
- Offline evaluation
Where you learn it
MMLU is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.