SWE-bench Verified
Princeton NLP (with OpenAI)
500 real GitHub issues, human-checked, used to score whether AI can fix bugs in popular Python projects.
A human-validated subset of the SWE-bench test set. One row is a GitHub issue paired with the repository snapshot and the unit tests the fix must pass. Evaluation runs the tests on a model's proposed patch.
Access
- Cost
- FreeNo charge.
- License
- License not statedNot stated in the Hub record; underlying repositories keep their own licenses
- Sign-up
- No sign-up
- How you get it
- API and Bulk downloadParquet
What's in it
- One row is
- One row per issue and pull-request pair
- Covers
- Not geographic
- Years
- Issues from open-source Python projects (see paper)
- Updated
- One-time release
- Size
- 500 tasks
Quality
Human screening removed unsolvable tasks, but the benchmark is Python-only and widely contaminated.
- Only a dozen or so repositories
- Models may have seen the repositories and fixes in training
- Scores sensitive to the agent scaffold
Connects to
- Terminal-Bench: Similar agentic evaluation focused on shell tasks
- GAIA: General-assistant benchmark for comparison
Questions it could answer
- Which repositories are solved most often?
- How does patch size relate to success?