MMLU-Pro
TIGER-Lab (University of Waterloo and collaborators)
A harder MMLU with ten answer options and 12,000 reasoning-heavy questions.
A benchmark that builds on MMLU by removing trivial questions, adding more reasoning problems, and expanding choices from four to ten. One row is a multiple-choice question with its options, answer, category, and source. It is among the most downloaded LLM evaluation sets.
Access
- Cost
- FreeNo charge.
- License
- Open licenseMIT
- Sign-up
- No sign-up
- How you get it
- API and Bulk downloadParquet
What's in it
- One row is
- One row per multiple-choice question
- Covers
- Not geographic
- Years
- 2024
- Updated
- Irregular
- Size
- About 12,000 questions
Quality
More discriminating than MMLU and well adopted, though it inherits some original MMLU errors.
- Some questions carry label errors from source datasets
- Public test set, so contamination is possible
- Multiple-choice format favors test-taking strategy
Connects to
Questions it could answer
- Which categories lose the most accuracy relative to MMLU?
- Do chain-of-thought answers help more on certain subjects?