What’s Out There
← Back to swiping
EducationFreeNonprofit

Humanity's Last Exam

Center for AI Safety and Scale AI

Thousands of expert-written questions designed to be too hard for current AI, across many fields.

A multimodal benchmark of closed-ended academic questions submitted by experts and kept only if frontier models failed them. One row is a question, possibly with an image, plus its answer and subject. The authors ask that the data not be reshared in crawlable form to protect it from training contamination.

Access

Cost
FreeFree but gated; accept the access terms on the Hub.
License
Open licenseMIT
Sign-up
Free account
How you get it
API and Bulk downloadParquet

What's in it

One row is
One row per question
Covers
Not geographic
Years
2024-2025
Updated
Irregular
Size
Between 1,000 and 10,000 questions
questionimageansweranswer typesubjectrationale

Quality

4 of 5

Credible and widely cited frontier eval, but still being revised after reviewers flagged problems.

  • Questions were filtered against specific models
  • Some answers have been reported as wrong and fixed in later versions
  • Gating exists to limit training on it

Connects to

  • GPQA: Earlier graduate-level science benchmark now nearly saturated
  • MMLU-Pro: Broad multiple-choice benchmark HLE aims to be harder than

Questions it could answer

  • Which subjects have the lowest model accuracy?
  • How do text-only and image questions compare?

Similar data sets

  • GAIA

    450 questions that are simple for people but need browsing, tools, and multi-step reasoning for AI assistants.

    Internet & techGAIA benchmark team (Meta, Hugging Face, and AutoGPT collaborators)Free

  • MMLU-Pro

    A harder MMLU with ten answer options and 12,000 reasoning-heavy questions.

    EducationTIGER-Lab (University of Waterloo and collaborators)Free

  • GPQA

    Graduate-level science questions where outside experts score only 34 percent with Google.

    EducationDavid Rein and collaborators (NYU and others)Free

  • NaturalReasoning

    Millions of hard questions back-translated from pretraining text and decontaminated.

    Language & textMeta (Facebook AI Research)Free

  • SWE-bench Verified

    500 real GitHub issues, human-checked, used to score whether AI can fix bugs in popular Python projects.

    Internet & techPrinceton NLP (with OpenAI)Free

  • SpreadsheetBench

    912 real Excel-forum questions, each tested on multiple spreadsheet files, to measure LLM spreadsheet skills.

    Internet & techRenmin University of China (RUC)Free