C4 Colossal Clean Crawled Corpus
Source, license and coverage
Supplier documentation. These claims are separate from the automated sample score. A listing edit date is not a data freshness date.
- License
- odc-by
- Source / creator
- allenai/c4
- Collection method
- Common Crawl WET text extracts were filtered with heuristic rules: keep only lines ending in terminal punctuation, drop pages with fewer than 5 sentences, retain only lines with at least 3 words, remove pages containing tokens from a profanity/blocklist ("List of Dirty, Naughty, Obscene or Otherwise Bad Words"), drop pages with the word "javascript", drop pages with placeholder text like "lorem ipsum" or curly braces (likely code), and deduplicate three-sentence spans across the corpus. Language detection (for mC4) uses cld3 with a 70% confidence threshold per page. AllenAI's redistribution mirrors Google's TFDS preparation.
- Coverage start
- Not documented
- Coverage end
- Not documented
- Data last updated
- Not documented
- Update schedule
- Not documented
- Snapshot is from April 2019 Common Crawl — does not reflect post-2019 web content; stale for current events, recent code, modern model outputs. - Heuristic cleaning still leaves significant boilerplate, SEO spam, and machine-generated text; documented studies (Dodge et al. 2021) show substantial overlap with copyrighted material, NLP benchmark contamination (GLUE, SQuAD, etc.), and demographic skew (over-representation of US/UK English, under-representation of African American English and minority dialects due to blocklist filtering). - mC4 language labels are noisy for low-resource languages; `und` (undetermined) subset is large. - Blocklist filtering disproportionately removes LGBTQ+ and minority-identity content from the `en` variant. - No PII scrubbing — URLs and document text may contain personal information.
Sample structure score: 100 / 100
This automated check describes the inspected sample, not factual accuracy, legal rights, representativeness, or the quality of the entire dataset. It is not a customer rating.
Assessed 10 sample records (JSON) on 2026-10-09. All records in the provided sample were checked.
| Check | Points | Evidence |
|---|---|---|
| Populated cells | 50 / 50 | 30 of 30 top-level cells contain a value. Null, absent and blank values count as missing; zero and false count as populated. |
| Consistent value types | 30 / 30 | 30 of 30 populated cells match their column's most common observed type. Types are inferred, not checked against real-world truth. |
| Consistent record shape | 20 / 20 | 10 of 10 records have the expected fields. CSV/TSV use the header width; JSON uses the union of observed keys. |
Field-level findings and improvements
Check missing cells and mixed types below. Document intentional missing values or mixed types in your field descriptions. Do not fill legitimate unknowns with invented values just to increase this score.
| Field | Missing cells | Most common type | Other populated types |
|---|---|---|---|
| text | 0 / 10 | string | 0 / 10 |
| timestamp | 0 / 10 | object | 0 / 10 |
| url | 0 / 10 | string | 0 / 10 |
About this data
Cleaned Common Crawl web text corpus spanning English and 108 additional languages. Foundational pretraining dataset used in T5 and numerous open language models.
Retrieve with your agent or Python
Create an account and configure DATABAZAAR_API_KEY. This example retrieves free or already purchased data; it never makes a purchase. For a multi-file dataset, choose a file index from the manifest.
Download the Python examplepython3 retrieve-dataset.py f1fd12f0-2418-4e09-b6ab-213dcc187974 --output dataset.bin
Full supplier documentation
Schema
| Name | Type | Description |
|---|---|---|
| text | VARCHAR | string — the cleaned document text extracted from a web page |
| timestamp | TIMESTAMP | string — Common Crawl fetch timestamp (ISO 8601) |
| url | VARCHAR | string — source URL of the document |
Sample Data
Preview a sample of the data before downloading.
Public sample only. Sign in to retrieve the full dataset, including free datasets.
For AI Agents
# 1. Add to your agent's MCP config (claude_desktop_config.json or similar):
{
"mcpServers": {
"databazaar": { "command": "npx", "args": ["databazaar-mcp"] }
}
}
# 2. Your agent can then call:
search_datasets({ query: "C4 Colossal Clean Crawled Corp" })
// Found: f1fd12f0-2418-4e09-b6ab-213dcc187974
get_download_url({ dataset_id: "f1fd12f0-2418-4e09-b6ab-213dcc187974" }) // free — sign in with MCP OAuth first# Free dataset — sign in or use your account API key: curl https://api.databazaar.io/datasets/f1fd12f0-2418-4e09-b6ab-213dcc187974/download-url -H "Authorization: Bearer $DATABAZAAR_API_KEY"