Humanity's Last Exam
Center for AI Safety and Scale AI
Thousands of expert-written questions designed to be too hard for current AI, across many fields.
A multimodal benchmark of closed-ended academic questions submitted by experts and kept only if frontier models failed them. One row is a question, possibly with an image, plus its answer and subject. The authors ask that the data not be reshared in crawlable form to protect it from training contamination.
Access
- Cost
- FreeFree but gated; accept the access terms on the Hub.
- License
- Open licenseMIT
- Sign-up
- Free account
- How you get it
- API and Bulk downloadParquet
What's in it
- One row is
- One row per question
- Covers
- Not geographic
- Years
- 2024-2025
- Updated
- Irregular
- Size
- Between 1,000 and 10,000 questions
Quality
Credible and widely cited frontier eval, but still being revised after reviewers flagged problems.
- Questions were filtered against specific models
- Some answers have been reported as wrong and fixed in later versions
- Gating exists to limit training on it
Connects to
Questions it could answer
- Which subjects have the lowest model accuracy?
- How do text-only and image questions compare?