Falcon RefinedWeb Corpus
Source, license and coverage
Supplier documentation. These claims are separate from the automated sample score. A listing edit date is not a data freshness date.
- License
- odc-by
- Source / creator
- tiiuae/falcon-refinedweb
- Collection method
- RefinedWeb is built from CommonCrawl WARC files using the Macrodata Refinement (MDR) pipeline described in the paper. The pipeline applies URL filtering against blocklists, language identification (English-only), trafilatura-based text extraction, document- and line-level heuristic quality filters (adapted from Gopher and C4), and large-scale fuzzy deduplication (MinHash) plus exact substring deduplication. The public release is a ~500B-token extract of the full 5T-token internal corpus.
- Coverage start
- Not documented
- Coverage end
- Not documented
- Data last updated
- Not documented
- Update schedule
- Not documented
- English-only; no multilingual coverage. - Time coverage ends with the CommonCrawl dumps used at construction (through early 2023); dataset is static and not refreshed. - Despite filtering, web data inevitably contains biased, toxic, PII-adjacent, or copyrighted material — TII does not claim full removal. - This is only ~10% of the full RefinedWeb corpus used internally by TII; downstream model quality may differ from the original Falcon training run. - Not deduplicated against common LLM evaluation benchmarks — benchmark contamination is plausible.
Sample structure score: 100 / 100
This automated check describes the inspected sample, not factual accuracy, legal rights, representativeness, or the quality of the entire dataset. It is not a customer rating.
Assessed 10 sample records (JSON) on 2026-10-09. All records in the provided sample were checked.
| Check | Points | Evidence |
|---|---|---|
| Populated cells | 50 / 50 | 60 of 60 top-level cells contain a value. Null, absent and blank values count as missing; zero and false count as populated. |
| Consistent value types | 30 / 30 | 60 of 60 populated cells match their column's most common observed type. Types are inferred, not checked against real-world truth. |
| Consistent record shape | 20 / 20 | 10 of 10 records have the expected fields. CSV/TSV use the header width; JSON uses the union of observed keys. |
Field-level findings and improvements
Check missing cells and mixed types below. Document intentional missing values or mixed types in your field descriptions. Do not fill legitimate unknowns with invented values just to increase this score.
| Field | Missing cells | Most common type | Other populated types |
|---|---|---|---|
| content | 0 / 10 | string | 0 / 10 |
| url | 0 / 10 | string | 0 / 10 |
| timestamp | 0 / 10 | object | 0 / 10 |
| dump | 0 / 10 | string | 0 / 10 |
| segment | 0 / 10 | number | 0 / 10 |
| image_urls | 0 / 10 | object | 0 / 10 |
About this data
English web corpus derived from CommonCrawl through stringent filtering and deduplication. Comprises approximately 968 million documents used to train the Falcon language model family.
Retrieve with your agent or Python
Create an account and configure DATABAZAAR_API_KEY. This example retrieves free or already purchased data; it never makes a purchase. For a multi-file dataset, choose a file index from the manifest.
Download the Python examplepython3 retrieve-dataset.py e44da883-fe75-4faf-88ba-9ecf2125b883 --output dataset.bin
Full supplier documentation
Schema
| Name | Type | Description |
|---|---|---|
| content | VARCHAR | string — the cleaned document text |
| url | VARCHAR | string — source URL of the page |
| timestamp | TIMESTAMP | string — CommonCrawl snapshot timestamp |
| dump | VARCHAR | string — CommonCrawl dump identifier (e.g. CC-MAIN-2023-06) |
| segment | VARCHAR | string — CommonCrawl WARC segment |
| image_urls | VARCHAR[][] | list — extracted image links with alt text (multimodal-friendly) |
Sample Data
Preview a sample of the data before downloading.
Public sample only. Sign in to retrieve the full dataset, including free datasets.
For AI Agents
# 1. Add to your agent's MCP config (claude_desktop_config.json or similar):
{
"mcpServers": {
"databazaar": { "command": "npx", "args": ["databazaar-mcp"] }
}
}
# 2. Your agent can then call:
search_datasets({ query: "Falcon RefinedWeb Corpus" })
// Found: e44da883-fe75-4faf-88ba-9ecf2125b883
get_download_url({ dataset_id: "e44da883-fe75-4faf-88ba-9ecf2125b883" }) // free — sign in with MCP OAuth first# Free dataset — sign in or use your account API key: curl https://api.databazaar.io/datasets/e44da883-fe75-4faf-88ba-9ecf2125b883/download-url -H "Authorization: Bearer $DATABAZAAR_API_KEY"