BigCodeBench Code Generation Benchmark

## Overview BigCodeBench is a code generation benchmark from the BigCode project containing 1,140 hand-crafted Python programming tasks across two variants: Complete (docstring-based code completion) and Instruct (natural-language instruction-based generation). Each task ships with executable unit tests averaging 5.6 test cases per task and 99% branch coverage. Format is Parquet; total dataset is in the 1K–10K row range across splits. ## Schema - task_id — string — unique identifier for the task - complete_prompt — string — full docstring-based prompt for the Complete variant - instruct_prompt — string — natural-language instruction prompt for the Instruct variant - canonical_solution — string — reference solution in Python - code_prompt — string — function signature / code-only prompt stub - test — string — Python unittest code used to grade generations - entry_point — string — function name to be implemented - doc_struct — string/json — structured representation of the docstring - libs — string — libraries used / required by the task ## Sources - bigcode/bigcodebench on Hugging Face — https://huggingface.co/datasets/bigcode/bigcodebench — license: Apache-2.0 - Paper: BigCodeBench (arXiv:2406.15877) ## Methodology Tasks were expert-authored by the BigCode collaboration to evaluate practical programming ability with diverse library usage (the benchmark intentionally exercises 100+ Python libraries across domains). Each task was paired with hand-written unit tests and verified to achieve ~99% branch coverage on the canonical solution. The Instruct variant was derived by rewriting docstring-style prompts into natural-language task instructions while preserving the same underlying test harness. ## Known gaps & limitations The benchmark is Python-only and skewed toward library-API-heavy tasks; it is not a general competitive-programming or algorithmic benchmark. As a public, widely-downloaded eval set, contamination risk in modern code LLM training corpora is non-trivial — the BigCode team has published contamination analyses but buyers training on this data should treat it as held-out eval, not training data. Test cases, while high-coverage, may still admit reward-hacking solutions that pass tests without genuinely solving the task. ## Intended use & out-of-scope - IS for: evaluating code-generation LLMs, agent coding benchmarks, prompt-engineering research, RAG-for-code evaluation, fine-tuning data only when explicitly held out from eval. - NOT for: training data for models that will be evaluated on BigCodeBench (leakage), non-Python code generation, or as a representative sample of real-world software engineering workloads. _Federated dataset: 5 parquet shards, 11.3 MB total. Queries and downloads stream through the DataBazaar API._ _PII signals: credit_card_candidate×2 (0.3≤score<0.7) present in the sample. Common in public datasets (papers, logs) but worth knowing before joining with private data._ Original supplier listing: BigCodeBench — Code Generation Benchmark (Complete & Instruct) 1,140-task code generation benchmark from BigCode with both docstring-based completion and NL-instruction variants, 99% test coverage, Apache-2.0 licensed.