FineWeb-Edu Translated Multilingual Corpus

## Overview Helsinki-NLP/fineweb-edu-translated is a massive multilingual corpus produced by automatically translating documents from the English FineWeb-Edu dataset into 36 European languages using OPUS-MT and HPLT-MT models. Version 1.0 covers 36,704,000 source documents and over 28 billion space-separated English tokens, expanding to more than 960 billion tokens of translated content. Documents are aligned across all language pairs, enabling parallel-corpus use cases. Stored in Parquet format. v1.1 adds further translations. ## Schema - document_id — string — stable identifier shared across language splits to enable alignment - text — string — translated (or original English) document content - language — string — ISO 639-3 language code (e.g. deu, fra, spa, ukr) - source — string — translation model used (OPUS-MT or HPLT-MT) - (additional FineWeb-Edu provenance fields may be present per language split) ## Sources - Helsinki-NLP/fineweb-edu-translated on HuggingFace — https://huggingface.co/datasets/Helsinki-NLP/fineweb-edu-translated — license: ODC-By - Upstream: HuggingFaceFW/fineweb-edu (ODC-By) - Translation models: OPUS-MT and HPLT-MT (Helsinki-NLP) ## Methodology Source documents come from FineWeb-Edu, an educational-quality filter applied to the FineWeb Common Crawl derivative. Helsinki-NLP runs each English document through OPUS-MT or HPLT-MT neural machine translation models to produce 36 target-language versions. Documents retain alignment via shared IDs so the same source content can be retrieved in any language, making the corpus usable as parallel data. No human post-editing is performed. ## Known gaps & limitations All non-English content is machine-translated and will contain MT artifacts, hallucinations, and fluency/adequacy errors typical of OPUS-MT/HPLT-MT — quality varies substantially by target language (high-resource Romance/Germanic languages generally better than low-resource Baltic, Celtic, or Balkan languages). Translation errors compound for technical, scientific, or culturally specific educational content. Coverage is restricted to 36 European languages — no Asian, African, or indigenous languages. Inherits any topical, source, or quality biases of the underlying FineWeb-Edu English filter. Not deduplicated against common multilingual eval suites. ## Intended use & out-of-scope - Intended for: multilingual LLM pretraining, continued pretraining for European languages, MT model training (as synthetic parallel data), cross-lingual transfer research, language-specific fine-tuning where native data is scarce. - Out of scope: ground-truth translation evaluation (it's MT output, not gold), benchmark training without leakage checks against FLORES/WMT/etc., production translation reference data, claims about native-speaker fluency. _Federated dataset: 2,742 parquet shards, 5062.25 GB total. Queries and downloads stream through the DataBazaar API._ Original supplier listing: FineWeb-Edu Translated (36 Languages, 960B+ Tokens) Machine-translated FineWeb-Edu corpus covering 36.7M aligned documents across 36 European languages — 960B+ tokens for multilingual LLM pretraining and translation training.