textScaleAI/SWE-bench_Proswe-benchcoding-agentsbenchmarkevaluationsoftware-engineeringscale-aiagentslong-horizoncodellm-eval

SWE-bench Pro

Free

Open dataset

Sample structure: 100 / 100

14 download links issued

Seller: DataBazaar

Already have an account? Log in

Agent? Connect your account →

Source, license and coverage

Supplier documentation. These claims are separate from the automated sample score. A listing edit date is not a data freshness date.

License: Not documented — confirm reuse terms with the seller
Source / creator: huggingface: ScaleAI/SWE-bench_Pro
Collection method: uploaded
Coverage start: Not documented
Coverage end: Not documented
Data last updated: Not documented
Update schedule: Not documented

Source documentation ↗

Sample structure score: 100 / 100

This automated check describes the inspected sample, not factual accuracy, legal rights, representativeness, or the quality of the entire dataset. It is not a customer rating.

Assessed 10 sample records (JSON) on 2026-10-09. All records in the provided sample were checked.

Check	Points	Evidence
Populated cells	50 / 50	160 of 160 top-level cells contain a value. Null, absent and blank values count as missing; zero and false count as populated.
Consistent value types	30 / 30	160 of 160 populated cells match their column's most common observed type. Types are inferred, not checked against real-world truth.
Consistent record shape	20 / 20	10 of 10 records have the expected fields. CSV/TSV use the header width; JSON uses the union of observed keys.

Field-level findings and improvements

Check missing cells and mixed types below. Document intentional missing values or mixed types in your field descriptions. Do not fill legitimate unknowns with invented values just to increase this score.

Field	Missing cells	Most common type	Other populated types
repo	0 / 10	string	0 / 10
instance_id	0 / 10	string	0 / 10
base_commit	0 / 10	string	0 / 10
patch	0 / 10	string	0 / 10
test_patch	0 / 10	string	0 / 10
problem_statement	0 / 10	string	0 / 10
requirements	0 / 10	string	0 / 10
interface	0 / 10	string	0 / 10
repo_language	0 / 10	string	0 / 10
fail_to_pass	0 / 10	string	0 / 10
pass_to_pass	0 / 10	string	0 / 10
issue_specificity	0 / 10	string	0 / 10
issue_categories	0 / 10	string	0 / 10
before_repo_set_cmd	0 / 10	string	0 / 10
selected_test_files_to_run	0 / 10	string	0 / 10
dockerhub_tag	0 / 10	string	0 / 10

How the score is calculated, its limitations, and how to correct an assessment →

About this data

Enterprise-level benchmark for evaluating AI agents on long-horizon software engineering tasks. Contains challenging real-world coding problems following SWE-Bench Verified structure.

Retrieve with your agent or Python

Create an account and configure DATABAZAAR_API_KEY. This example retrieves free or already purchased data; it never makes a purchase. For a multi-file dataset, choose a file index from the manifest.

Download the Python example

python3 retrieve-dataset.py de2141e6-1a79-421b-a389-801739457e65 --output dataset.bin

Full supplier documentation

SWE-bench Pro is a challenging, enterprise-level evaluation dataset designed by Scale AI for testing agent capabilities on long-horizon software engineering tasks. It follows the structure of SWE-Bench Verified but raises the difficulty bar with more complex, realistic problems representative of enterprise software engineering work. This dataset is ideal for: - Evaluating coding agents and SWE assistants - Benchmarking long-horizon reasoning and tool use - Comparing agent performance on enterprise-grade tasks - Research on autonomous software engineering The dataset is in parquet format, well-structured, and ships with an open-source evaluation harness on GitHub. With ~37k+ downloads and high industry visibility from Scale AI, it represents a key benchmark in the agentic coding evaluation landscape. Paper: https://static.scale.com/uploads/654197dc94d34f66c0f5184e/SWEAP_Eval_Scale%20(9).pdf Evaluation harness: https://github.com/scaleapi/SWE-bench_Pro-os Original supplier listing: SWE-bench Pro Enterprise-level benchmark dataset from Scale AI for evaluating AI agents on long-horizon software engineering tasks. Follows SWE-Bench Verified structure with challenging real-world coding problems.

Schema

Name	Type	Description
repo	VARCHAR	Repository owner and name (e.g., 'NodeBB/NodeBB')
instance_id	VARCHAR	Unique identifier for the task instance combining repo, commit hash, and variant
base_commit	VARCHAR	Git commit hash representing the initial state before the fix
patch	VARCHAR	Unified diff format showing the complete solution changes required
test_patch	VARCHAR	Unified diff format for test file modifications needed to validate the fix
problem_statement	VARCHAR	Natural language description of the software engineering issue to resolve
requirements	VARCHAR	Specific constraints, dependencies, or implementation requirements for the task
interface	VARCHAR	API signatures, function definitions, or class interfaces that must be implemented
repo_language	VARCHAR	Primary programming language of the repository (e.g., 'JavaScript', 'Python')
fail_to_pass	VARCHAR	Test identifiers or commands that must transition from failing to passing state
pass_to_pass	VARCHAR	Test identifiers or commands that must remain passing throughout the solution
issue_specificity	VARCHAR	Categorical level describing precision of the problem scope (e.g., 'high', 'medium')
issue_categories	VARCHAR	Comma-separated tags classifying issue type (e.g., 'bug', 'feature', 'refactor')
before_repo_set_cmd	VARCHAR	Shell command(s) to execute setup or initialization in the repository context
selected_test_files_to_run	VARCHAR	Comma-separated paths to test files used for validating the solution
dockerhub_tag	VARCHAR	Docker image reference specifying environment and dependencies for evaluation

Sample Data

Preview a sample of the data before downloading.

Public sample only. Sign in to retrieve the full dataset, including free datasets.

For AI Agents

Via MCP Server

# 1. Add to your agent's MCP config (claude_desktop_config.json or similar):
{
  "mcpServers": {
    "databazaar": { "command": "npx", "args": ["databazaar-mcp"] }
  }
}

# 2. Your agent can then call:
search_datasets({ query: "SWE-bench Pro" })
// Found: de2141e6-1a79-421b-a389-801739457e65
get_download_url({ dataset_id: "de2141e6-1a79-421b-a389-801739457e65" })  // free — sign in with MCP OAuth first

Via REST API

# Free dataset — sign in or use your account API key:
curl https://api.databazaar.io/datasets/de2141e6-1a79-421b-a389-801739457e65/download-url -H "Authorization: Bearer $DATABAZAAR_API_KEY"