What’s Out There
← Back to swiping
EducationFreeAcademic

MMLU-Pro

TIGER-Lab (University of Waterloo and collaborators)

A harder MMLU with ten answer options and 12,000 reasoning-heavy questions.

A benchmark that builds on MMLU by removing trivial questions, adding more reasoning problems, and expanding choices from four to ten. One row is a multiple-choice question with its options, answer, category, and source. It is among the most downloaded LLM evaluation sets.

Access

Cost
FreeNo charge.
License
Open licenseMIT
Sign-up
No sign-up
How you get it
API and Bulk downloadParquet

What's in it

One row is
One row per multiple-choice question
Covers
Not geographic
Years
2024
Updated
Irregular
Size
About 12,000 questions
questionoptionsanswercategorysourcechain-of-thought explanation

Quality

4 of 5

More discriminating than MMLU and well adopted, though it inherits some original MMLU errors.

  • Some questions carry label errors from source datasets
  • Public test set, so contamination is possible
  • Multiple-choice format favors test-taking strategy

Connects to

  • MMLU: Original benchmark this one extends
  • GPQA: Harder science-only evaluation
  • MMMLU: MMLU translated into 14 languages

Questions it could answer

  • Which categories lose the most accuracy relative to MMLU?
  • Do chain-of-thought answers help more on certain subjects?

Similar data sets

  • MMMLU (Multilingual MMLU)

    The MMLU test set translated into 14 languages by professional human translators.

    EducationOpenAIFree

  • Humanity's Last Exam

    Thousands of expert-written questions designed to be too hard for current AI, across many fields.

    EducationCenter for AI Safety and Scale AIFree

  • NaturalReasoning

    Millions of hard questions back-translated from pretraining text and decontaminated.

    Language & textMeta (Facebook AI Research)Free

  • GPQA

    Graduate-level science questions where outside experts score only 34 percent with Google.

    EducationDavid Rein and collaborators (NYU and others)Free

  • Llama-Nemotron Post-Training Dataset

    Millions of math, code, and reasoning examples used to post-train NVIDIA's Llama-Nemotron models.

    Language & textNVIDIAFree

  • SpreadsheetBench

    912 real Excel-forum questions, each tested on multiple spreadsheet files, to measure LLM spreadsheet skills.

    Internet & techRenmin University of China (RUC)Free