FineWeb2 Multilingual Web Corpus

## Overview FineWeb2 is a massive multilingual web pretraining corpus from HuggingFace covering over 1,000 languages, derived from CommonCrawl snapshots and processed with language-specific filtering and deduplication pipelines. The dataset contains billions of documents (1B<n<10B rows), distributed as parquet shards organized by language. It is the successor to the popular English-focused FineWeb dataset and is intended as a foundation for training multilingual LLMs. ## Schema - `text` — string — main document text content - `id` — string — unique document identifier - `dump` — string — CommonCrawl dump identifier (e.g., CC-MAIN-2024-XX) - `url` — string — source URL of the document - `date` — string — crawl timestamp - `file_path` — string — path within the CommonCrawl WARC archive - `language` — string — detected language code (ISO 639-3) - `language_score` — float — confidence of language detection - `language_script` — string — detected writing script - `minhash_cluster_size` — int — deduplication cluster size - `top_langs` — string — JSON of top detected language candidates ## Sources - HuggingFaceFW/fineweb-2 on HuggingFace: https://huggingface.co/datasets/HuggingFaceFW/fineweb-2 — License: ODC-By 1.0 - Upstream: CommonCrawl (https://commoncrawl.org) — also ODC-By aligned - Reference papers: arXiv:2506.20920 (FineWeb2), arXiv:2406.17557 (FineWeb), arXiv:2109.07445 (OSCAR-related methodology) ## Methodology FineWeb2 starts from raw CommonCrawl WARC files, applies language identification (GlotLID and similar classifiers), then performs per-language quality filtering, MinHash near-deduplication, and PII redaction. Processing decisions were guided by hundreds of ablation experiments on 9 diverse pilot languages, and the full pipeline (datatrove) is open source and reproducible. Each language partition is shipped as parquet shards with metadata columns preserved for downstream filtering. ## Known gaps & limitations Quality varies dramatically across the 1000+ languages — high-resource languages (English, Mandarin, Spanish, Hindi) have far more data and stricter filter validation than low-resource languages where ablation studies were not run. Language identification errors are non-trivial for closely related languages and code-switched content. Web-sourced content includes biases toward commercial, technical, and Western perspectives; some toxic, NSFW, or factually incorrect material remains despite filtering. Temporal coverage is tied to CommonCrawl snapshots and skews recent. Not aligned to any specific eval benchmark — leakage risk if used naively for evaluation training. ## Intended use & out-of-scope - **Intended for:** Pretraining and continued pretraining of multilingual LLMs, language-specific model training (especially for low-resource languages), tokenizer training, linguistic research on web text distributions. - **Not for:** Production use without further filtering for the user's safety requirements; benchmark evaluation (not deduplicated against common eval suites — leakage risk); authoritative source of facts; assumed-clean PII-free corpus despite redaction efforts. _Federated dataset: 4,569 parquet shards, 8070.11 GB total. Queries and downloads stream through the DataBazaar API._ _PII signals: cc_shape×5 (Luhn-valid: 0) present in the sample. Common in public datasets (papers, logs) but worth knowing before joining with private data._ Original supplier listing: FineWeb2: Multilingual Web Pretraining Corpus (1000+ Languages) Second iteration of HuggingFace's FineWeb dataset, covering 1000+ languages with high-quality filtered web text for LLM pretraining. ODC-By 1.0 licensed, validated through hundreds of ablation experiments.