SWE-bench
Verified
500 manually verified real software engineering tasks created by the Princeton NLP team, sourced from real GitHub repositories' bug fixes and feature requests, serve as the gold standard benchmark set for evaluating AI code agents (such as Claude Code, Devin, SWE-Agent).
Dataset Highlights
The authoritative benchmark for evaluating AI code agents' capabilities, with each task meticulously verified by humans.
Real GitHub Issues
Each task comes from real GitHub repository Issues, containing complete problem descriptions, contextual information, and code change history, reflecting real software development scenarios.
Human-Verified Tasks
All 500 tasks have been individually reviewed and verified by humans to ensure clear problem descriptions, correct solutions, and effective test patches, eliminating noise from automated filtering.
Test Patch Verification
Each task comes with a corresponding test patch (Test Patch) that can automatically verify whether the AI-generated code fixes pass all relevant test cases correctly.
12 Major Python Repositories
Covers well-known open-source projects such as Django, Flask, scikit-learn, sympy, matplotlib, spanning multiple fields including web frameworks, scientific computing, and data visualization.
Code Agent Benchmark
Industry-recognized evaluation standard for AI code agents, with cutting-edge systems like Claude Code, Devin, SWE-Agent, and OpenDevin measured against this benchmark.
MIT License
Utilizes the MIT open-source license, freely usable for academic research, commercial evaluation, and product development, with no usage restrictions, and an active community for continuous updates.
Applicable Scenarios
Comprehensive assessment of AI programming capabilities from academic research to engineering practice
AI Code Agent Evaluation
Measure the performance of AI programming assistants like Claude Code, Devin, SWE-Agent, etc., in real software engineering tasks
Software Engineering Research
Provide standardized evaluation datasets for research directions such as automated program repair, code generation, and defect localization
Automated Bug Fixing
Evaluate and train automated defect repair systems, validating their generalization capabilities across projects and domains
Development Tool Benchmarking
Provide standardized performance evaluation schemes for IDE plugins, code review tools, and CI/CD systems
Data Preview
The following is a single task example structure from the SWE-bench Verified dataset
{
"instance_id": "django__django-16379",
"repo": "django/django",
"base_commit": "a1e4e3f...",
"problem_statement": "FileBasedCache has_key is susceptible to race conditions...",
"hints_text": "",
"created_at": "2022-12-04T17:43:25Z",
"patch": "diff --git a/django/core/cache/backends/filebased.py ...",
"test_patch": "diff --git a/tests/cache/tests.py ...",
"version": "5.0",
"FAIL_TO_PASS": ["test_has_key_race_handling (cache.tests.FileBasedCacheTests)"],
"PASS_TO_PASS": ["test_cache_key_i18n (cache.tests.FileBasedCacheTests)", "..."],
"environment_setup_commit": "4c76ffc..."
}
3 Steps to Get Started Quickly
Start in minutes from data acquisition to evaluating the AI agent
Browse Datasets
View dataset details on the Ace Data Cloud platform to understand metadata such as task structure, coverage repository, and licensing agreements.
Get Data
Obtain 500 validation tasks via API, each containing a problem description, code patch, and test patch, ready to use.
Run Evaluation
Compare the patches generated by the AI agent with test_patch for validation, assess the pass rate, and generate performance reports.
Start Evaluating AI Code Agents
500 manually verified real tasks, MIT open license, available immediately. Whether you are a researcher or an engineer, this benchmark dataset is the preferred choice for evaluating AI programming capabilities.