MMMLU (Multilingual MMLU)
OpenAI
The MMLU test set translated into 14 languages by professional human translators.
OpenAI's translation of the 57-subject MMLU test set, covering topics from elementary knowledge to professional law and physics. One row is one question in one language. It lets researchers measure whether a model's knowledge holds across languages, using human rather than machine translation.
Access
- Cost
- FreeNo charge.
- License
- Open licenseMIT
- Sign-up
- No sign-up
- How you get it
- API and Bulk downloadCSV, Parquet
What's in it
- One row is
- One row per translated question
- Covers
- 14 languages worldwide
- Years
- 2024
- Updated
- One-time release
- Size
- MMLU test set (about 14,000 questions) in each of 14 languages
Quality
Professional translation is a strength; the content still inherits MMLU's errors and Western-centric subjects.
- Inherits label errors from MMLU
- Translated, not culturally adapted
- Public test set
Connects to
- MMLU-Pro: Harder English benchmark from the same family
- FLEURS: Another multilingual benchmark, for speech
Questions it could answer
- How large is the accuracy gap between English and lower-resource languages?
- Which subjects lose the most in translation?