textSWE-bench/SWE-benchswe-benchcode-generationagentsbenchmarkgithub-issuespythonevaluationllm-evalcoding-agentsfine-tuning
SWE-bench: Real-World GitHub Issue Resolution Benchmark
About this data
2,294 Issue-PR pairs from 12 popular Python repos for evaluating LLM/agent ability to resolve real GitHub issues. Verified via post-PR unit tests. Canonical agent coding benchmark.
Schema
| Name | Type | Description |
|---|---|---|
| repo | VARCHAR | GitHub repository identifier in owner/name format |
| instance_id | VARCHAR | Unique identifier for the issue-PR pair |
| base_commit | VARCHAR | Git commit SHA of repository state before fix |
| patch | VARCHAR | Unified diff of the gold-standard fix (PR changes) |
| test_patch | VARCHAR | Unified diff containing unit tests for verification |
| problem_statement | VARCHAR | Full text of the GitHub issue describing the bug or feature request |
| hints_text | VARCHAR | Additional discussion comments from the issue thread |
| created_at | VARCHAR | ISO 8601 timestamp when the issue was created |
| version | VARCHAR | Repository version tag or release identifier |
| FAIL_TO_PASS | VARCHAR | Test cases that fail before fix and pass after fix |
| PASS_TO_PASS | VARCHAR | Test cases that pass both before and after fix |
| environment_setup_commit | VARCHAR | Commit SHA used to set up the test environment |
Sample Data
Preview a sample of the data before downloading.
Free
Open dataset
Quality: No ratings
1 downloads
Seller: DataBazaar
Agent? No sign-up needed →
For AI Agents
Via MCP Server
# 1. Add to your agent's MCP config (claude_desktop_config.json or similar):
{
"mcpServers": {
"databazaar": { "command": "npx", "args": ["databazaar-mcp"] }
}
}
# 2. Your agent can then call:
search_datasets({ query: "SWE-bench: Real-World GitHub I" })
// Found: c8dddd6d-99ec-4934-9396-43455b088bd8
get_download_url({ dataset_id: "c8dddd6d-99ec-4934-9396-43455b088bd8" }) // free — no API key neededVia REST API
# Free dataset — no API key required: curl https://api.databazaar.io/datasets/c8dddd6d-99ec-4934-9396-43455b088bd8/download-url