FineWeb-Edu Educational Web Content Corpus

## Overview FineWeb-Edu is a 1.3 trillion token English-language corpus of educational web content, distilled from the larger FineWeb dataset (which itself was derived from CommonCrawl snapshots covering 2013–2024). Content was filtered using an educational-quality classifier trained on annotations from Llama3-70B-Instruct. The dataset is distributed as Parquet files (billions of rows) and is one of the most-downloaded pretraining corpora on Hugging Face (400k+ downloads). ## Schema - `text` — string — the cleaned web page text content - `id` — string — unique document identifier - `dump` — string — CommonCrawl dump identifier (e.g., CC-MAIN-2024-10) - `url` — string — source URL of the document - `date` — string — crawl date - `file_path` — string — path within the CommonCrawl WARC archive - `language` — string — detected language (en) - `language_score` — float — language detection confidence - `token_count` — int — token count (GPT-2 tokenizer) - `score` — float — educational quality classifier score - `int_score` — int — integer-bucketed educational score (used for filtering) ## Sources - HuggingFaceFW/fineweb-edu — https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu — license: ODC-By 1.0 - Underlying source: CommonCrawl (https://commoncrawl.org/) — CommonCrawl Terms of Use - Paper: https://arxiv.org/abs/2406.17557 ## Methodology The HuggingFace FineWeb team began with 96 CommonCrawl snapshots, applied URL filtering, language identification, quality and repetition heuristics from the FineWeb pipeline, and MinHash deduplication. They then trained a linear classifier on top of Snowflake-arctic-embed embeddings using 450k Llama3-70B-Instruct annotations rating educational value of web samples on a 0–5 scale. Documents scoring ≥3 were retained, yielding the 1.3T-token FineWeb-Edu subset. A larger 5.4T variant (score ≥2) is also published separately. ## Known gaps & limitations - English-only — non-English educational content is excluded - "Educational" is defined by a Llama3-derived classifier and reflects the biases of that model's judgments - Source is CommonCrawl, so coverage is biased toward indexable, crawlable web pages; paywalled and non-public educational material is absent - PII has been filtered with regexes but the team notes residual PII may remain - Not deduplicated against common LLM evaluation benchmarks — direct use for pretraining models that will be evaluated on MMLU, ARC, etc. carries leakage risk - Crawl coverage skews toward recent years; older web content is underrepresented ## Intended use & out-of-scope - IS for: LLM pretraining and continued pretraining, especially for models targeting reasoning and knowledge tasks; RAG corpus construction; classifier training data; large-scale text analysis - NOT for: benchmark evaluation training without leakage analysis; applications requiring guaranteed PII removal; non-English use cases; tasks needing curated/verified factual accuracy (this is filtered web text, not authoritative educational material) _Federated dataset: 5,446 parquet shards, 9647.08 GB total. Queries and downloads stream through the DataBazaar API._ _PII signals: cc_shape×40 (Luhn-valid: 0) present in the sample. Common in public datasets (papers, logs) but worth knowing before joining with private data._ Original supplier listing: FineWeb-Edu: 1.3T Tokens of Educational Web Content 1.3 trillion tokens of high-quality educational web pages filtered from FineWeb using a Llama3-70B-trained classifier. Parquet format, ODC-By licensed, ideal for LLM pretraining and RAG.