Sci-Base Scientific Foundation Dataset

## Overview Sci-Base is a multi-domain scientific text dataset from OpenDataLab, part of the broader Sciverse data foundation aimed at the AI for Science (AI4S) community. It aggregates scientific content across chemistry, biology, climate, medical, materials, earth science, and physics. The corpus is sized in the 1M–10M row range, distributed as Parquet files, and is English-language text modality. ## Schema Full schema is defined per Parquet partition on the HuggingFace dataset page. Typical columns in OpenDataLab scientific corpora include: - `text` — string — primary scientific passage / document body - `domain` — string — subject area (chem, bio, climate, medical, material, earth, physics) - `source` — string — upstream source identifier - `id` — string — record identifier - additional metadata fields per domain (see HF dataset viewer for exact columns) ## Sources - HuggingFace: https://huggingface.co/datasets/opendatalab/Sci-Base — license: CC-BY-4.0 - Publisher: OpenDataLab (Shanghai AI Lab), as part of the Sciverse initiative ## Methodology Per the publisher, Sci-Base is curated as part of Sciverse, a multi-layered scientific data foundation designed to supply high-quality data resources for building scientific knowledge systems and accelerating AI4S research. Content is aggregated and normalized across multiple scientific domains into a unified Parquet schema suitable for large-scale pretraining and retrieval workloads. Specific per-source collection and filtering pipelines are described on the HuggingFace dataset card. ## Known gaps & limitations Source does not exhaustively document per-domain coverage balance, deduplication against scientific benchmarks (e.g., MMLU-STEM, SciQ, PubMedQA), or upstream license heterogeneity at the record level; buyers should validate empirically before use in evaluation-adjacent training. English-only modality tag — non-English scientific literature is out of scope. Domain distribution across the seven listed fields is not guaranteed to be balanced. ## Intended use & out-of-scope - **Intended:** Pretraining and continued pretraining of scientific LLMs, RAG corpora for AI4S agents, domain-adaptive fine-tuning across chemistry/biology/climate/medical/materials/earth/physics, retrieval index construction. - **Out-of-scope:** Not deduplicated against common scientific eval suites — direct training risks benchmark leakage; not a substitute for primary literature access; not validated for clinical or safety-critical medical decision-making. _Federated dataset: 218 parquet shards, 217.88 GB total. Queries and downloads stream through the DataBazaar API._ _PII signals: cc_shape×4 (Luhn-valid: 0), email×19, us_phone×3 present in the sample. Common in public datasets (papers, logs) but worth knowing before joining with private data._ Original supplier listing: Sci-Base: AI-Ready Scientific Foundation Dataset Large multi-domain scientific text corpus (chem, bio, climate, medical, materials, earth, physics) from OpenDataLab's Sciverse foundation, 1M-10M rows in Parquet, CC-BY-4.0.