What’s Out There
← Back to swiping
Internet & techFreeCommunity project

Every Eval Ever Datastore

EvalEval Coalition (community)

One schema for AI evaluation results scraped from leaderboards, papers, and local runs, so they can be joined.

A community database of model evaluation results. One row is one JSON record per model per evaluation run, which may carry several scores. It standardizes results that otherwise sit in scattered leaderboards and papers, and is updated frequently.

Access

Cost
FreeNo charge.
License
Open licenseMIT
Sign-up
No sign-up
How you get it
API and Bulk downloadJSON, Parquet

What's in it

One row is
One record per model per evaluation run
Covers
Not geographic
Years
2020s to the present
Updated
Irregular
Size
1,000 to 10,000 records in the record's size category
model namebenchmarkscoressource (leaderboard, paper, or local run)evaluation settingssample-level results

Quality

3 of 5

Useful idea with a defined schema, but scraped from heterogeneous sources and still growing.

  • Scores from different harnesses are not strictly comparable
  • Coverage is uneven across models and benchmarks
  • Scraped results can contain transcription errors

Connects to

  • Arena Leaderboard Dataset: Human-preference rankings that complement benchmark scores
  • MMLU-Pro: One of the benchmarks whose results are recorded
  • GAIA: Agent benchmark with results collected here

Questions it could answer

  • How closely do benchmark rankings agree with each other across models?
  • Which benchmarks have saturated, and when?

Similar data sets

  • Arena Leaderboard Dataset

    Historical snapshots of the Arena leaderboard, showing how model rankings moved over time.

    Internet & techLMArenaFree

  • Chatbot Arena Conversations

    33,000 side-by-side model chats where real users voted for the better answer.

    Language & textLMSYS (UC Berkeley and collaborators)Free

  • Cooperative Election Study (CES)

    Tens of thousands of Americans each election year, enough to study opinion inside individual states.

    Public opinionHarvard University (with YouGov)Free

  • Humanity's Last Exam

    Thousands of expert-written questions designed to be too hard for current AI, across many fields.

    EducationCenter for AI Safety and Scale AIFree

  • GAIA

    450 questions that are simple for people but need browsing, tools, and multi-step reasoning for AI assistants.

    Internet & techGAIA benchmark team (Meta, Hugging Face, and AutoGPT collaborators)Free

  • Synthetic Text-to-SQL

    105,851 synthetic text-to-SQL records spanning dozens of domains and SQL task types.

    Internet & techGretel AIFree