GAIA
GAIA benchmark team (Meta, Hugging Face, and AutoGPT collaborators)
450 questions that are simple for people but need browsing, tools, and multi-step reasoning for AI assistants.
A benchmark of real-world assistant tasks with unambiguous answers, some with attached files such as audio, images, and documents. One row is a question with its level, file, and final answer. The validation set has answers; the test set answers are withheld for a leaderboard, and access is gated.
Access
- Cost
- FreeFree but gated to prevent bot scraping.
- License
- License not statedNot stated in the Hub record
- Sign-up
- Free account
- How you get it
- API and Bulk downloadJSON Lines, Parquet
What's in it
- One row is
- One row per question
- Covers
- Not geographic
- Years
- 2023
- Updated
- One-time release
- Size
- More than 450 questions (fewer than 1,000)
Quality
A respected agent benchmark with human-written questions; small size and live web dependence limit reproducibility.
- Some answers depend on web pages that change
- Do not reshare in crawlable form
- Small, so scores are noisy
Connects to
- SWE-bench Verified: Agent benchmark focused on software issues instead of web tasks
- Terminal-Bench: Agent benchmark for command-line tasks
Questions it could answer
- Which levels expose the biggest gap between humans and agents?
- Which attached-file types cause the most failures?