GPQA
David Rein and collaborators (NYU and others)
Graduate-level science questions where outside experts score only 34 percent with Google.
Multiple-choice questions written and validated by domain experts, designed to be search-resistant. One row is a question with one correct and three incorrect answers. Several subsets exist, including a harder Diamond set. The authors ask users not to reveal examples in plain text online.
Access
- Cost
- FreeFree but gated; accept terms on the Hub.
- License
- Open licenseCC BY 4.0
- Sign-up
- Free account
- How you get it
- API and Bulk downloadCSV
What's in it
- One row is
- One row per multiple-choice question
- Covers
- Not geographic
- Years
- 2023
- Updated
- One-time release
- Size
- Between 1,000 and 10,000 rows across subsets (the Diamond set is smaller)
Quality
Carefully built with expert validation and a clear contamination request; small size means noisy comparisons.
- Small test sets give wide confidence intervals
- Top models now score near the ceiling
- Contamination risk if leaked online
Connects to
- Humanity's Last Exam: Harder successor benchmark
- MMLU-Pro: Broader and easier general-knowledge evaluation
Questions it could answer
- Which subdomain is hardest for models versus experts?
- How does accuracy change when the expert-validated answers are used?