MATH-500 Benchmark

## Overview MATH-500 is a 500-problem subset of the Hendrycks MATH benchmark, curated by OpenAI for the 'Let's Verify Step by Step' paper and redistributed by HuggingFace H4. It is the de-facto standard evaluation set for measuring mathematical reasoning capability in modern LLMs and reasoning models (o1, R1, etc.). The dataset is small (500 rows), text-only English, JSON-formatted, and covers competition-level problems across algebra, geometry, number theory, counting & probability, intermediate algebra, prealgebra, and precalculus, each with difficulty levels 1–5 and a ground-truth solution. ## Schema - problem — string — the math problem statement (LaTeX-formatted) - solution — string — full worked solution with final answer in \boxed{} - answer — string — extracted final answer - subject — string — topic category (e.g. Algebra, Geometry, Number Theory) - level — int — difficulty 1 (easiest) to 5 (hardest) - unique_id — string — stable identifier referencing original MATH test split path ## Sources - HuggingFaceH4/MATH-500 on HuggingFace: https://huggingface.co/datasets/HuggingFaceH4/MATH-500 (MIT, inherited from upstream) - Upstream: OpenAI PRM800K repo, https://github.com/openai/prm800k (MIT license) - Original benchmark: Hendrycks et al., 'Measuring Mathematical Problem Solving With the MATH Dataset' (2021), MIT license ## Methodology Problems were originally scraped from US high-school math competitions (AMC, AIME, etc.) by Hendrycks et al. and annotated with step-by-step solutions in LaTeX. OpenAI then sampled 500 problems from the MATH test split to form a stable, smaller evaluation subset for their process-reward-model work; the selection is published in PRM800K. HuggingFace H4 mirrors that exact subset as JSON with no transformations beyond format conversion. ## Known gaps & limitations This is a small evaluation set (n=500), not a training corpus. It is widely used and almost certainly contaminated in the pretraining data of most frontier LLMs, so scores should be interpreted as upper-bound capability indicators rather than held-out generalization. Coverage is English-only, US competition-math style, and skews toward symbolic manipulation rather than applied or proof-based math. Answers are extracted from \boxed{} expressions, so grading requires careful normalization (equivalent forms, LaTeX whitespace). The source does not document selection bias in how OpenAI chose the 500. ## Intended use & out-of-scope - IS for: evaluating mathematical reasoning of LLMs, reasoning-model benchmarking (o1/R1-style), RLHF/RLVR reward signal validation, prompt-engineering ablations. - NOT for: training data (high leakage risk into evals), non-English math evaluation, real-world applied math, or proof-generation tasks. Original supplier listing: MATH-500 (HuggingFaceH4) 500-problem subset of the MATH benchmark used by OpenAI in 'Let's Verify Step by Step' — the canonical math reasoning eval for LLMs and reasoning models.