SWE-bench GitHub Issue Resolution Benchmark

## Overview SWE-bench is the canonical benchmark for evaluating language models and agents on resolving real-world GitHub issues. It contains 2,294 Issue–Pull Request pairs collected from 12 popular Python repositories (e.g., django, sympy, scikit-learn, matplotlib, flask, requests, pytest, sphinx, astropy, xarray, pylint, pyodide). Each instance includes the issue text, the target repository state, and the unit tests that validate a correct fix. Evaluation uses post-PR test behavior as the reference solution. Format: Parquet, ~10K–100K rows across splits (dev/test), text modality. ## Schema - instance_id — string — unique identifier for the issue/PR pair - repo — string — owner/name of the source GitHub repository - base_commit — string — commit SHA of the repo state before the fix - patch — string — the gold patch (PR diff) that resolves the issue - test_patch — string — diff containing the unit tests used for verification - problem_statement — string — the issue text describing the bug or feature request - hints_text — string — additional discussion/comments from the issue thread - created_at — string — issue creation timestamp - version — string — repository version tag for environment setup - FAIL_TO_PASS — string (JSON list) — tests expected to flip from fail→pass after the fix - PASS_TO_PASS — string (JSON list) — tests that should continue passing - environment_setup_commit — string — commit used to install the testing environment ## Sources - HuggingFace: https://huggingface.co/datasets/SWE-bench/SWE-bench — license: CC BY 4.0 (per the dataset card / project repo) - Paper: "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" (Jimenez et al., arXiv:2310.06770) - Upstream project: https://github.com/princeton-nlp/SWE-bench ## Methodology The authors mined merged pull requests from 12 widely-used Python repositories on GitHub, filtering to PRs that (a) closed an issue, (b) modified source code, and (c) added or modified unit tests. For each candidate, they reconstructed the repository state at the base commit, applied the test patch, and verified that the gold code patch caused a specified set of tests to transition from failing to passing (FAIL_TO_PASS) while not breaking previously passing tests (PASS_TO_PASS). Each instance is bundled with metadata sufficient to reproduce a Python environment for execution-based evaluation. No content rewriting is applied; problem_statements are the raw issue text. ## Known gaps & limitations - Python-only and limited to 12 repositories — does not represent broader software engineering work (no C/C++, JS, Go, Java, etc.). - Heavy weighting toward scientific/web Python ecosystem; agent performance on these may not generalize. - Known contamination concerns: many frontier LLMs were trained on the underlying public GitHub repos, so issues and their fixes may appear in pretraining data — leakage risk for any model with a training cutoff after the PR merge date. - This base split contains only problem_statement and metadata; for execution you typically need SWE-bench_Lite, SWE-bench_Verified, or the companion Docker images / harness from the upstream repo. - Hints_text may contain spoilers (maintainer-written fix suggestions) that make some tasks easier than a realistic from-scratch issue triage. - Time coverage: PRs span the historical lifetime of the included repos through ~2023; not refreshed with newer issues. ## Intended use & out-of-scope - Intended: evaluating coding agents, LLM patch generation, retrieval-augmented code fix systems, agent planning/tool-use benchmarks, and SFT/RL fine-tuning on agentic coding trajectories. - Out-of-scope: training a model and then reporting numbers on this same split without decontamination — leakage against GitHub-trained LLMs is well-documented. For leaderboard-quality eval use SWE-bench_Verified. _Federated dataset: 3 parquet shards, 114.4 MB total. Queries and downloads stream through the DataBazaar API._ _PII signals: email×8 present in the sample. Common in public datasets (papers, logs) but worth knowing before joining with private data._ Original supplier listing: SWE-bench: Real-World GitHub Issue Resolution Benchmark 2,294 Issue-PR pairs from 12 popular Python repos for evaluating LLM/agent ability to resolve real GitHub issues. Verified via post-PR unit tests. Canonical agent coding benchmark.