LongCoT Long-Horizon Reasoning Benchmark

## Overview LongCoT is a benchmark designed to evaluate whether language models can sustain coherent reasoning across long chains of thought. It spans five domains — logic, computer science, chemistry, chess, and mathematics — and is distributed in Parquet format for easy loading via the HuggingFace `datasets` library, pandas, or polars. The release contains between 1K and 10K examples and is intended for use as an evaluation harness rather than as training data. ## Schema Schema is not fully documented on the dataset card; based on the task category (question-answering) and the benchmark intent, expect columns covering: - `question` / `prompt` — text — the reasoning problem statement - `answer` / `solution` — text — the reference answer used by the verifier - `domain` / `category` — string — one of logic, cs, chemistry, chess, math - `metadata` — struct/text — problem-specific fields (e.g. difficulty, source) Buyers should consult the HF dataset viewer for exact column names per split. ## Sources - HuggingFace: https://huggingface.co/datasets/LongHorizonReasoning/longcot — license: MIT - Canonical code, verifier, and eval harness: https://github.com/LongHorizonReasoning/longcot - Associated paper: arXiv:2604.14140 ## Methodology LongCoT was constructed by the LongHorizonReasoning group to measure sustained reasoning quality. Problems are drawn or synthesized across five domains chosen because they admit programmatic verification (e.g., chess legality, chemical structure checks, math/CS answer checking). The accompanying GitHub repo provides the canonical verifier and evaluation harness; the HF release is a viewer-friendly Parquet mirror of the benchmark data. ## Known gaps & limitations The dataset card on HuggingFace is brief and does not exhaustively document construction, per-domain sample counts, or contamination checks against common pretraining corpora — the tags mention contamination-detection but specifics are not detailed in the card. Coverage is English-only. Buyers running evals on frontier models should validate that test items have not leaked into training sets and should consult the upstream GitHub for the canonical verifier rather than relying on string-match scoring. ## Intended use & out-of-scope - Intended: evaluation of long-horizon / long-CoT reasoning capabilities in LLMs; ablation studies on chain-of-thought length; verifier-based scoring across multiple reasoning domains. - Out-of-scope: training or fine-tuning data (benchmark contamination risk); single-domain specialist evaluation where domain-native benchmarks (e.g., MATH, HumanEval, ChessBench) are more appropriate. _Federated dataset: 18 parquet shards, 13.6 MB total. Queries and downloads stream through the DataBazaar API._ ## Temporal validity This dataset includes column(s) keyed on recycled identifiers — the same value can refer to different entities at different times: - **domain name** (reissued by registrars (drop-catching)) — domains are recycled; use WHOIS history to filter enrichment by the current registration interval. Original supplier listing: LongCoT: Long-Horizon Reasoning Benchmark Benchmark for evaluating sustained long chain-of-thought reasoning across logic, computer science, chemistry, chess, and mathematics. Parquet format, MIT licensed.