Wikimedia Wikipedia Multilingual Corpus

## Overview A cleaned, article-level dump of Wikipedia covering 300+ language editions, built directly from the official Wikimedia dumps (https://dumps.wikimedia.org/). Each language is a separate subset with a single `train` split; each row is one full article with markdown/wikitext stripped and non-content sections (references, external links, etc.) removed. Total scale is tens of millions of articles across all languages combined (10M–100M rows per HF size category). Format is Parquet. Snapshot built from late-2023 dumps and published Jan 2024. ## Schema - `id` — string — Wikipedia page ID - `url` — string — canonical URL of the article on the corresponding language Wikipedia - `title` — string — article title - `text` — string — cleaned plain-text body of the article ## Sources - wikimedia/wikipedia on Hugging Face — https://huggingface.co/datasets/wikimedia/wikipedia — licensed CC-BY-SA-3.0 and GFDL (dual) - Upstream: Wikimedia Foundation dumps — https://dumps.wikimedia.org/ ## Methodology The dataset is produced from the official Wikimedia XML dumps. The Wikimedia team parses wikitext, strips markup, removes structural/non-prose sections (references, see-also, external links, navigation templates), and emits one record per article. Per-language subsets are processed uniformly and packaged as Parquet for streaming-friendly access. No additional re-ranking, deduplication across languages, or quality filtering is applied beyond the markup cleanup. ## Known gaps & limitations - Snapshot is from late-2023 dumps published Jan 2024 — any article edits, new articles, or deletions after that date are not reflected. - Cleaning is heuristic: some templates, infobox content, tables, and math/markup residue may leak into `text`, while some legitimate prose can be stripped. - Article quality varies enormously across languages — large editions (en, de, fr, es, ru, zh, ja) are well-developed; many small-language editions are stubs or machine-translated and should not be treated as high-quality NLP signal. - No deduplication against common LLM eval suites (MMLU, TriviaQA, NaturalQuestions, etc.) — substantial leakage risk if used to train models you then benchmark. - CC-BY-SA-3.0 / GFDL require attribution and share-alike on derivative distributions; downstream model weights trained on this corpus are generally treated as out of scope of share-alike, but redistributing the text itself requires attribution. ## Intended use & out-of-scope - Intended: LLM pretraining and continued pretraining, multilingual language modeling, RAG knowledge bases, entity/knowledge extraction, retrieval eval construction, fine-tuning data mixing. - Out-of-scope: as ground truth for factuality without verification; as a clean benchmark training set (eval leakage); as a real-time knowledge source (snapshot is stale by months-to-years). _Federated dataset: 555 parquet shards, 66.86 GB total. Queries and downloads stream through the DataBazaar API._ Original supplier listing: Wikimedia Wikipedia (All Languages, Cleaned) Cleaned full-text Wikipedia articles across 300+ language subsets, built from official Wikimedia dumps. Parquet format, one row per article. Foundational corpus for LLM pretraining, RAG, and multilingual NLP.