What’s Out There
← Back to swiping
Internet & techFreeAcademic

SWE-bench Verified

Princeton NLP (with OpenAI)

500 real GitHub issues, human-checked, used to score whether AI can fix bugs in popular Python projects.

A human-validated subset of the SWE-bench test set. One row is a GitHub issue paired with the repository snapshot and the unit tests the fix must pass. Evaluation runs the tests on a model's proposed patch.

Access

Cost
FreeNo charge.
License
License not statedNot stated in the Hub record; underlying repositories keep their own licenses
Sign-up
No sign-up
How you get it
API and Bulk downloadParquet

What's in it

One row is
One row per issue and pull-request pair
Covers
Not geographic
Years
Issues from open-source Python projects (see paper)
Updated
One-time release
Size
500 tasks
repositoryissue textbase commitgold patchtest patchfail-to-pass tests

Quality

4 of 5

Human screening removed unsolvable tasks, but the benchmark is Python-only and widely contaminated.

  • Only a dozen or so repositories
  • Models may have seen the repositories and fixes in training
  • Scores sensitive to the agent scaffold

Connects to

  • Terminal-Bench: Similar agentic evaluation focused on shell tasks
  • GAIA: General-assistant benchmark for comparison

Questions it could answer

  • Which repositories are solved most often?
  • How does patch size relate to success?

Similar data sets

  • Humanity's Last Exam

    Thousands of expert-written questions designed to be too hard for current AI, across many fields.

    EducationCenter for AI Safety and Scale AIFree

  • SpreadsheetBench

    912 real Excel-forum questions, each tested on multiple spreadsheet files, to measure LLM spreadsheet skills.

    Internet & techRenmin University of China (RUC)Free

  • GAIA

    450 questions that are simple for people but need browsing, tools, and multi-step reasoning for AI assistants.

    Internet & techGAIA benchmark team (Meta, Hugging Face, and AutoGPT collaborators)Free

  • The Public Jira Dataset

    2.7 million issues and 32 million changes from public Jira trackers, with comments and links.

    Internet & techMontgomery, Lüders, Maalej (University of Hamburg)Free

  • CVEfixes: vulnerabilities and their fixes from open-source software

    Security vulnerabilities linked to the exact code changes that fixed them.

    Internet & techMoonen & VidziunasFree

  • OpenR1-Math-220k

    220,000 math problems, each with two to four DeepSeek-R1 reasoning traces, many checked automatically.

    EducationHugging Face (Open-R1 project)Free