What’s Out There
← Back to swiping
EducationFreeAcademic

GPQA

David Rein and collaborators (NYU and others)

Graduate-level science questions where outside experts score only 34 percent with Google.

Multiple-choice questions written and validated by domain experts, designed to be search-resistant. One row is a question with one correct and three incorrect answers. Several subsets exist, including a harder Diamond set. The authors ask users not to reveal examples in plain text online.

Access

Cost
FreeFree but gated; accept terms on the Hub.
License
Open licenseCC BY 4.0
Sign-up
Free account
How you get it
API and Bulk downloadCSV

What's in it

One row is
One row per multiple-choice question
Covers
Not geographic
Years
2023
Updated
One-time release
Size
Between 1,000 and 10,000 rows across subsets (the Diamond set is smaller)
questioncorrect answerincorrect answerssubdomainexpert validation notes

Quality

4 of 5

Carefully built with expert validation and a clear contamination request; small size means noisy comparisons.

  • Small test sets give wide confidence intervals
  • Top models now score near the ceiling
  • Contamination risk if leaked online

Connects to

Questions it could answer

  • Which subdomain is hardest for models versus experts?
  • How does accuracy change when the expert-validated answers are used?

Similar data sets

  • Humanity's Last Exam

    Thousands of expert-written questions designed to be too hard for current AI, across many fields.

    EducationCenter for AI Safety and Scale AIFree

  • MMLU-Pro

    A harder MMLU with ten answer options and 12,000 reasoning-heavy questions.

    EducationTIGER-Lab (University of Waterloo and collaborators)Free

  • GAIA

    450 questions that are simple for people but need browsing, tools, and multi-step reasoning for AI assistants.

    Internet & techGAIA benchmark team (Meta, Hugging Face, and AutoGPT collaborators)Free

  • UltraFeedback

    64,000 prompts, four model answers each, scored by GPT-4 on four aspects.

    Language & textOpenBMB (Tsinghua University)Free

  • NaturalReasoning

    Millions of hard questions back-translated from pretraining text and decontaminated.

    Language & textMeta (Facebook AI Research)Free

  • MMMLU (Multilingual MMLU)

    The MMLU test set translated into 14 languages by professional human translators.

    EducationOpenAIFree