What’s Out There
← Back to swiping
EducationFreeCompany

MMMLU (Multilingual MMLU)

OpenAI

The MMLU test set translated into 14 languages by professional human translators.

OpenAI's translation of the 57-subject MMLU test set, covering topics from elementary knowledge to professional law and physics. One row is one question in one language. It lets researchers measure whether a model's knowledge holds across languages, using human rather than machine translation.

Access

Cost
FreeNo charge.
License
Open licenseMIT
Sign-up
No sign-up
How you get it
API and Bulk downloadCSV, Parquet

What's in it

One row is
One row per translated question
Covers
14 languages worldwide
Years
2024
Updated
One-time release
Size
MMLU test set (about 14,000 questions) in each of 14 languages
questionoptions A to Danswersubjectlanguage

Quality

4 of 5

Professional translation is a strength; the content still inherits MMLU's errors and Western-centric subjects.

  • Inherits label errors from MMLU
  • Translated, not culturally adapted
  • Public test set

Connects to

  • MMLU-Pro: Harder English benchmark from the same family
  • FLEURS: Another multilingual benchmark, for speech

Questions it could answer

  • How large is the accuracy gap between English and lower-resource languages?
  • Which subjects lose the most in translation?

Similar data sets

  • MMLU-Pro

    A harder MMLU with ten answer options and 12,000 reasoning-heavy questions.

    EducationTIGER-Lab (University of Waterloo and collaborators)Free

  • Multilingual parallel text corpora for East African languages

    English text translated into five East African languages, for training machine translation.

    Language & textMakerere University NLP groupFree

  • SmolLM-Corpus

    The mix of synthetic textbooks, educational web text, and Python code that trained small SmolLM models.

    Language & textHugging Face (SmolLM team)Free

  • Llama-Nemotron Post-Training Dataset

    Millions of math, code, and reasoning examples used to post-train NVIDIA's Llama-Nemotron models.

    Language & textNVIDIAFree

  • The Pile

    The 825 GB open training set that taught the field how LLM data is assembled, and got sued over it.

    Language & textEleutherAIFree

  • LMSYS-Chat-1M

    One million real conversations with 25 LLMs, collected from Vicuna demo and Chatbot Arena visitors.

    Language & textLMSYS (UC Berkeley and collaborators)Free