What’s Out There
← Back to swiping
Internet & techFreeCompany

GAIA

GAIA benchmark team (Meta, Hugging Face, and AutoGPT collaborators)

450 questions that are simple for people but need browsing, tools, and multi-step reasoning for AI assistants.

A benchmark of real-world assistant tasks with unambiguous answers, some with attached files such as audio, images, and documents. One row is a question with its level, file, and final answer. The validation set has answers; the test set answers are withheld for a leaderboard, and access is gated.

Access

Cost
FreeFree but gated to prevent bot scraping.
License
License not statedNot stated in the Hub record
Sign-up
Free account
How you get it
API and Bulk downloadJSON Lines, Parquet

What's in it

One row is
One row per question
Covers
Not geographic
Years
2023
Updated
One-time release
Size
More than 450 questions (fewer than 1,000)
questiondifficulty levelattached filefinal answerannotator steps

Quality

4 of 5

A respected agent benchmark with human-written questions; small size and live web dependence limit reproducibility.

  • Some answers depend on web pages that change
  • Do not reshare in crawlable form
  • Small, so scores are noisy

Connects to

  • SWE-bench Verified: Agent benchmark focused on software issues instead of web tasks
  • Terminal-Bench: Agent benchmark for command-line tasks

Questions it could answer

  • Which levels expose the biggest gap between humans and agents?
  • Which attached-file types cause the most failures?

Similar data sets

  • Humanity's Last Exam

    Thousands of expert-written questions designed to be too hard for current AI, across many fields.

    EducationCenter for AI Safety and Scale AIFree

  • UltraFeedback

    64,000 prompts, four model answers each, scored by GPT-4 on four aspects.

    Language & textOpenBMB (Tsinghua University)Free

  • GPQA

    Graduate-level science questions where outside experts score only 34 percent with Google.

    EducationDavid Rein and collaborators (NYU and others)Free

  • GDPval

    220 real work tasks across 44 occupations, with reference files, to test AI on economically valuable output.

    LaborOpenAIFree

  • HelpSteer2

    Prompt-response pairs rated on five attributes, small enough to inspect.

    Language & textNVIDIA (with Scale AI)Free

  • OpenAssistant Conversations (OASST1)

    161,443 chat messages written and ranked by volunteers in 35 languages, with no model in the loop.

    Language & textOpenAssistant (LAION-affiliated volunteers)Free