FinePDFs Multilingual Corpus

## Overview FinePDFs is the largest publicly available text corpus extracted exclusively from PDFs, containing approximately 3 trillion tokens spread across 475 million documents in 1,733 languages. Released by HuggingFaceFW (the team behind FineWeb), it complements web-crawl pretraining data with longer-form, more technical, and more linguistically diverse content typical of the PDF medium (academic papers, government docs, manuals, legal filings, books). Data is distributed as Parquet with tabular/text modality and is consumable via the `datasets`, `dask`, and `polars` libraries. ## Schema - `text` — string — the extracted PDF body text - `id` — string — document identifier - `language` — string — ISO 639-3 language code (one of 1,733) - `language_score` — float — language identification confidence - `url` / `source_url` — string — origin URL of the PDF - `dump` / `snapshot` — string — Common Crawl / source snapshot identifier - `token_count` — int — approximate token count per document - `extractor` — string — which extraction pipeline was used (Docling, OCR fallback, etc.) - +several more metadata columns (file hash, mime, page count, quality signals) ## Sources - HuggingFaceFW/finepdfs on Hugging Face — https://huggingface.co/datasets/HuggingFaceFW/finepdfs — license: ODC-By 1.0 - Underlying PDFs sourced from Common Crawl PDF captures - Associated papers: arXiv:2506.18421, arXiv:2109.07445 ## Methodology The HuggingFaceFW team identified PDFs in Common Crawl, ran them through layout-aware extraction (Docling and OCR fallbacks for image-only PDFs), language-identified each document, and applied quality filtering and deduplication consistent with the FineWeb family. Per-language token budgets were computed to surface the long tail of low-resource languages. Normalization includes text cleanup, boilerplate removal, and removal of documents with very low language-ID confidence. ## Known gaps & limitations - PDF extraction is inherently lossy: tables, equations, multi-column layouts, and figures may be garbled or dropped. - OCR-derived text for scanned/image PDFs contains recognition errors; quality varies sharply by language and script. - Long-tail languages (the bulk of the 1,733) have very small per-language token counts — useful for coverage but not for training competitive monolingual models. - PDFs over-represent academic, governmental, and legal genres; underrepresent conversational/colloquial text. - Deduplication is fuzzy across near-duplicate documents; some leakage with academic benchmarks (e.g., arXiv-derived eval sets) is likely. - Copyright status of individual PDFs is heterogeneous; the ODC-By license covers the compilation, not necessarily the underlying source documents — downstream users should perform their own diligence for commercial training. ## Intended use & out-of-scope - Intended: large-scale LLM pretraining, multilingual model training, long-context training, RAG corpora for technical/academic domains, linguistic research on low-resource languages. - Out-of-scope: NOT deduplicated against common eval suites (MMLU, GSM8K source materials, arXiv-based benchmarks) — benchmark contamination risk; NOT a clean source for low-resource MT without per-language quality auditing; NOT suitable as ground truth for PDF layout/structure tasks (extraction artifacts present). _Federated dataset: 3,525 parquet shards, 5006.14 GB total. Queries and downloads stream through the DataBazaar API._ _PII signals: cc_shape×5 (Luhn-valid: 0) present in the sample. Common in public datasets (papers, logs) but worth knowing before joining with private data._ Original supplier listing: FinePDFs — 3T Tokens from 475M PDFs in 1,733 Languages Largest public PDF-sourced corpus: ~3 trillion tokens, 475M documents, 1,733 languages. Parquet format, ODC-By licensed. Built by HuggingFaceFW for pretraining and multilingual research.