Common Corpus Multilingual Text

## Overview Common Corpus is the largest openly-licensed text corpus available, comprising approximately 2.27 trillion tokens (2,267,302,720,836) of text drawn from books, newspapers, scientific articles, government and legal documents, source code, and other permissibly-licensed sources. The dataset is distributed in Parquet format and spans 13+ languages including English, French, German, Chinese, Italian, Spanish, Japanese, Polish, Latin, Dutch, Russian, Arabic, and Korean. It was assembled by PleIAs in collaboration with partners and is intended as a fully-open alternative to mixed-license web crawl corpora for LLM pretraining. ## Schema - `text` — string — the document or passage content - `identifier` — string — source/document ID - `collection` — string — sub-corpus the document belongs to (e.g. cultural heritage, scientific, legal, code) - `language` — string — ISO language code - `license` — string — original license of the source document - `date` — string/int — publication or source date where available - `source` — string — upstream provider (e.g. Wikisource, PubMed, government archive) - `token_count` — int — approximate token count - +additional provenance and quality columns per sub-collection ## Sources - PleIAs/common_corpus on Hugging Face — https://huggingface.co/datasets/PleIAs/common_corpus — openly/permissibly licensed (per-document licenses retained; aggregate is open) - Reference paper: arXiv:2410.22587 (ICLR 2026 oral) - Underlying sources include public domain books, openly-licensed scientific literature (e.g. PubMed Central OA subset), government/legal documents, permissively-licensed code, and openly-licensed news. ## Methodology PleIAs aggregated text from sources whose licenses permit redistribution and downstream use — primarily public domain works, government documents, openly-licensed scientific publications, and permissively-licensed code. Per-document license metadata is retained so downstream users can filter by license type. The corpus has been normalized to a uniform Parquet schema, with language tagging, deduplication, and quality filtering applied by the curators. See the full paper for methodology details on filtering, OCR cleanup, and provenance tracking. ## Known gaps & limitations - Language balance is heavily skewed toward English and major European languages; lower-resource languages are underrepresented. - Significant portions are derived from older public-domain texts (pre-1928), which introduces archaic language, OCR artifacts, and outdated factual content. - OCR quality varies across cultural heritage sources; some documents contain recognition errors. - Per-document license metadata is retained but downstream users are responsible for honoring source-level license terms (e.g. attribution for CC-BY components). - Source does not exhaustively document deduplication against common evaluation benchmarks; buyers should validate empirically for eval contamination. ## Intended use & out-of-scope - **Intended:** LLM pretraining and continued pretraining where license cleanliness matters; research on open-data language modeling; multilingual model development; RAG corpora needing permissive provenance. - **Out-of-scope:** Not deduplicated against common eval suites (MMLU, HellaSwag, etc.) — leakage risk for benchmark training. Not a current-events corpus — heavy historical/public-domain skew makes it unsuitable as a sole source for recency-sensitive applications. _PII signals: cc_shape×2 (Luhn-valid: 0), email×1, us_phone×1 present in the sample. Common in public datasets (papers, logs) but worth knowing before joining with private data._ Original supplier listing: Common Corpus — 2.27T Tokens of Permissibly-Licensed Text Largest open and permissibly-licensed text dataset: 2.27 trillion tokens across books, newspapers, scientific articles, legal docs, code, and more in 13+ languages. Built by PleIAs for LLM pretraining.