Common Crawl Creative Commons Fine Multilingual Corpus

## Overview Common Crawl Creative Commons Fine (C5f) is a high-quality, multilingual web text corpus derived from Common Crawl by retaining only Creative Commons-licensed documents that also appear in the FineWeb or FineWeb-2 datasets. The dataset contains between 10M and 100M rows in Parquet format, covering eight languages (English, German, French, Dutch, Spanish, Italian, Afrikaans, West Frisian). Time coverage spans Common Crawl snapshots from CC-MAIN-2019-30 through CC-MAIN-2025-05. ## Schema - text — string — main document text content - url — string — source URL of the document - license — string — detected Creative Commons license variant - language — string — ISO language code - crawl — string — Common Crawl snapshot identifier (e.g., CC-MAIN-2024-51) - (additional metadata columns inherited from C5 source) ## Sources - BramVanroy/CommonCrawl-CreativeCommons-fine on HuggingFace (https://huggingface.co/datasets/BramVanroy/CommonCrawl-CreativeCommons-fine) — license: cc - Upstream: BramVanroy/CommonCrawl-CreativeCommons (C5) — filtered against HuggingFaceFW/fineweb and fineweb-2 ## Methodology The C5 base corpus was built by scanning Common Crawl WARC/WET files for documents that explicitly declare a Creative Commons license (via meta tags, RDFa, or link rel markers). The 'fine' variant further restricts the corpus to URLs that also passed FineWeb's quality filters (deduplication, language identification, heuristic quality scoring, and Gopher-style filtering). This intersection yields a smaller but higher-quality subset suitable for language modeling. ## Known gaps & limitations Coverage is limited to the eight listed languages, with strong skew toward English and Western European languages. Creative Commons license detection relies on declared metadata which may be incorrect or stale — buyers should validate license compliance for high-stakes use. Not all Common Crawl snapshots are included; coverage is sparse between 2020 and 2023. The FineWeb filter is English-optimized, so non-English quality filtering inherits FineWeb-2's separate methodology. Source does not document additional gaps; buyers should validate empirically. ## Intended use & out-of-scope - IS for: LLM pretraining and continued pretraining on permissively-licensed multilingual web text, RAG corpus construction, linguistic research on CC-licensed web content. - NOT for: applications requiring guaranteed license provenance without manual review; benchmark training without dedup against common eval suites (leakage risk); languages outside the eight covered. _Federated dataset: 728 parquet shards, 169.12 GB total. Queries and downloads stream through the DataBazaar API._ Original supplier listing: Common Crawl Creative Commons Fine (C5f) — Multilingual High-Quality Web Corpus Filtered Creative Commons web corpus from Common Crawl, intersected with FineWeb/FineWeb-2 for quality. Multilingual (EN, DE, FR, NL, ES, IT, AF, FY), 10M-100M rows, Parquet format. Ideal for LLM pretraining and fine-tuning.