ClimbMix English Pre-training Corpus

## Overview ClimbMix is a 400-billion-token English pre-training corpus released by NVIDIA (via OptimalScale) and introduced in arXiv:2504.13161. It is designed for efficient large language model pre-training, delivering strong downstream performance under an equal token budget compared to other open web corpora. The dataset is distributed in JSON format with hundreds of millions of documents (size category 100M<n<1B rows). ## Schema - `text` — string — the raw document text used for language model pre-training - Additional metadata fields may include topic group / cluster id and quality signals from the filtering pipeline (see source card for exact field names) The schema is intentionally minimal — this is a text-only pre-training corpus, not a structured dataset. ## Sources - Hugging Face: https://huggingface.co/datasets/OptimalScale/ClimbMix — License: Apache-2.0 - Paper: "ClimbMix" (arXiv:2504.13161) - Original publisher: NVIDIA, redistributed via OptimalScale ## Methodology Per the source description, the authors proposed a new algorithm to filter and mix web-scale pre-training data. Documents were first grouped into 1,000 topic clusters. Two classifiers were then applied: one to detect advertisements and another for additional quality filtering. The resulting topic-aware mixture was tuned to maximize per-token downstream task performance, yielding a compact 400B-token dataset that the authors report outperforms larger unfiltered corpora at equal training budgets. ## Known gaps & limitations - English-only — not suitable for multilingual pre-training without supplementation. - Derived from web crawl data; despite ad/quality filtering, residual boilerplate, toxicity, PII, and factual errors are likely present and not exhaustively audited. - Not deduplicated against common LLM evaluation suites — benchmark contamination risk should be assessed by the user. - Source does not document temporal coverage of underlying crawls; buyers should validate empirically for recency-sensitive use. - Topic clustering and classifier decisions may encode biases from their training data. ## Intended use & out-of-scope - IS for: LLM pre-training and continued pre-training, ablation studies on data mixing, research on quality filtering, large-scale tokenizer training. - NOT for: evaluation benchmarks (leakage risk), multilingual training, factual QA grounding without retrieval, or any setting requiring verified provenance per document. _Federated dataset: 10 parquet shards, 2.56 GB total. Queries and downloads stream through the DataBazaar API._ Original supplier listing: ClimbMix: 400B-Token Pre-training Corpus (NVIDIA) A 400-billion-token English pre-training corpus from NVIDIA, filtered and topic-mixed for efficient LLM pre-training with superior performance per token.