Every Eval Ever Datastore
EvalEval Coalition (community)
One schema for AI evaluation results scraped from leaderboards, papers, and local runs, so they can be joined.
A community database of model evaluation results. One row is one JSON record per model per evaluation run, which may carry several scores. It standardizes results that otherwise sit in scattered leaderboards and papers, and is updated frequently.
Access
- Cost
- FreeNo charge.
- License
- Open licenseMIT
- Sign-up
- No sign-up
- How you get it
- API and Bulk downloadJSON, Parquet
What's in it
- One row is
- One record per model per evaluation run
- Covers
- Not geographic
- Years
- 2020s to the present
- Updated
- Irregular
- Size
- 1,000 to 10,000 records in the record's size category
Quality
Useful idea with a defined schema, but scraped from heterogeneous sources and still growing.
- Scores from different harnesses are not strictly comparable
- Coverage is uneven across models and benchmarks
- Scraped results can contain transcription errors
Connects to
- Arena Leaderboard Dataset: Human-preference rankings that complement benchmark scores
- MMLU-Pro: One of the benchmarks whose results are recorded
- GAIA: Agent benchmark with results collected here
Questions it could answer
- How closely do benchmark rankings agree with each other across models?
- Which benchmarks have saturated, and when?