CodeParrot Clean — Deduplicated Python Code

## Overview CodeParrot Clean is a large corpus of Python source files scraped from public GitHub repositories, cleaned and deduplicated for use in code language model pretraining and fine-tuning. The dataset contains roughly 5M+ Python files (1M<n<10M size class) in JSON format with content and repository metadata. Snapshot reflects GitHub state as of the original CodeParrot collection (circa 2022). ## Schema - `content` — string — raw Python source code of the file - `repo_name` — string — GitHub repository identifier (owner/repo) - `path` — string — file path within the repository - `copies` — int — number of near-duplicate copies observed before dedup - `size` — int — file size in bytes - `license` — string — detected license of the source repo (where available) - `hash` — string — content hash used for deduplication - `line_mean` / `line_max` — float/int — line length statistics used in filtering - `alpha_frac` — float — fraction of alphanumeric characters - `autogenerated` — bool — heuristic flag for autogenerated code ## Sources - HuggingFace: https://huggingface.co/datasets/codeparrot/codeparrot-clean — original source files scraped from public GitHub via BigQuery / GH Archive. - Upstream parent: codeparrot/codeparrot (raw, undeduplicated). - License: dataset card does not assert a single uniform license; underlying files retain their original repo licenses. Aggregated dataset is distributed as research/open access on HF. ## Methodology The CodeParrot team queried public GitHub for Python files, then applied the following cleaning pipeline: (1) exact-match deduplication on file content hash; (2) filter to files with average line length < 100 and max line length < 1000; (3) require alphanumeric character fraction > 0.25; (4) drop files matching keywords associated with auto-generated code. The result is a cleaner, lower-redundancy corpus than the raw scrape, oriented toward training code generation models. ## Known gaps & limitations - Snapshot is from 2022; does not reflect recent Python ecosystem changes (e.g., newer libraries, typing patterns, recent frameworks). - Per-file license metadata is best-effort from repo-level detection; downstream users must respect individual file licenses for derivative works. - Filtering heuristics (alphanumeric fraction, auto-generated keyword list) are imperfect — some boilerplate and generated code remain. - Python only; not suitable for multi-language code modeling without augmentation. - Likely overlap with common code benchmarks (HumanEval, MBPP) — leakage risk if used to train models evaluated on those suites. ## Intended use & out-of-scope - **Intended:** pretraining/fine-tuning code LMs for Python, RAG over Python code, static analysis research, code-style and pattern mining. - **Out-of-scope:** training models evaluated on HumanEval/MBPP without contamination checks; production use without reviewing the per-file licenses of redistributed snippets. _Federated dataset: 10 parquet shards, 1.69 GB total. Queries and downloads stream through the DataBazaar API._ _PII signals: cc_shape×10 (Luhn-valid: 0), email×3, us_phone×17 present in the sample. Common in public datasets (papers, logs) but worth knowing before joining with private data._ Original supplier listing: CodeParrot Clean — Deduplicated Python Code from GitHub Cleaned, deduplicated corpus of Python files scraped from GitHub. Filtered for line length, alphanumeric fraction, and auto-generated content. Widely used for code LM pretraining and fine-tuning.