ChemBench Chemistry & Materials LLM Evaluation Benchmark

## Overview ChemBench is a manually curated benchmark for evaluating the chemistry and materials science capabilities of Large Language Models. The dataset contains between 1K and 10K expert-generated items in English, covering question-answering, multiple-choice, language modeling, and natural language inference task formats. Distributed as parquet, it is intended strictly as an evaluation suite — the maintainers explicitly warn against using it for training or fine-tuning. Associated paper: arXiv:2411.16955. ## Schema - question — string — the chemistry/materials prompt or stem - choices — list/string — answer options for multiple-choice items - answer — string — the gold/reference answer - topic — string — chemistry subdomain or task category - task_type — string — question-answering vs multiple-choice vs NLI - source — string — origin or curator attribution for the item - difficulty / metadata — string — additional curation metadata - (schema varies by subset; see HF dataset card for full breakdown) ## Sources - jablonkagroup/ChemBench on Hugging Face — https://huggingface.co/datasets/jablonkagroup/ChemBench — License: MIT - Paper: Mirza et al., "Are large language models superhuman chemists?" arXiv:2411.16955 ## Methodology Items were manually authored and curated by domain experts (annotations_creators: expert-generated, language_creators: expert-generated) to probe chemistry and materials science reasoning, including factual recall, calculation, safety, and structured problem-solving. The source dataset is original (not derived from other corpora) and monolingual English. Items are organized into multiple task formats to enable both open-ended generation and constrained multiple-choice scoring. ## Known gaps & limitations English-only; coverage is breadth-oriented across chemistry subfields but the source does not exhaustively document subdomain balance — buyers running narrow evaluations (e.g., specific reaction classes) should validate coverage empirically. Because the benchmark is widely circulated, contamination risk in modern frontier models is non-trivial; treat scores on this set with the usual eval-leakage caveats. Maintainers explicitly prohibit use of these items for training/fine-tuning, as doing so invalidates the benchmark. ## Intended use & out-of-scope - IS for: zero-shot / few-shot evaluation of LLM chemistry and materials reasoning, leaderboarding, regression testing of model releases, scientific-domain capability audits. - NOT for: training, fine-tuning, RLHF reward modeling, or any data-augmentation use that would leak gold answers into model weights — doing so violates the dataset's stated intent and corrupts the benchmark. _Federated dataset: 9 parquet shards, 0.5 MB total. Queries and downloads stream through the DataBazaar API._ Original supplier listing: ChemBench — Chemistry & Materials LLM Evaluation Benchmark Manually curated benchmark for evaluating chemistry and materials science capabilities of LLMs. Expert-generated QA and multiple-choice items. MIT licensed, evaluation-only.