MMLU-Pro Benchmark

## Overview MMLU-Pro is a more robust and challenging successor to MMLU, designed to rigorously benchmark large language model reasoning across academic and professional disciplines. It contains approximately 12,000 multiple-choice questions spanning 14 subject categories (math, physics, chemistry, law, engineering, psychology, health, business, biology, philosophy, economics, history, computer science, and others). Each question has up to 10 answer options (vs. MMLU's 4), reducing the chance of correct guessing and increasing the reasoning demand. Distributed as parquet files via Hugging Face. ## Schema - `question_id` — int — unique identifier for each question - `question` — string — the question prompt - `options` — list[string] — up to 10 candidate answer choices - `answer` — string — the correct option letter (A–J) - `answer_index` — int — zero-based index of the correct option - `cot_content` — string — chain-of-thought reasoning trace (where provided) - `category` — string — subject category (e.g., math, law, physics) - `src` — string — original source of the question (e.g., original MMLU, STEM textbooks, TheoremQA, SciBench) ## Sources - TIGER-Lab/MMLU-Pro on Hugging Face — https://huggingface.co/datasets/TIGER-Lab/MMLU-Pro — MIT license - Paper: "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark" (arXiv:2406.01574) ## Methodology The authors built MMLU-Pro by (1) filtering the original MMLU to retain only questions that remain difficult for current frontier models, (2) augmenting with reasoning-heavy questions sourced from STEM textbooks, TheoremQA, and SciBench, and (3) expanding the answer choice set from 4 to up to 10 distractors per question to reduce guessing. Questions were reviewed for correctness and difficulty, and chain-of-thought reasoning content was added for many items to enable few-shot CoT evaluation. ## Known gaps & limitations - English-only; no multilingual coverage. - Heavily skewed toward STEM and academic disciplines; underrepresents practical/applied or culturally specific knowledge. - Widely used as a public benchmark — high risk of training-data contamination in newer frontier models. Buyers using this for fine-tuning should expect leakage into common pretraining corpora. - Some questions inherit any errors or ambiguities from the original MMLU and source textbooks. - Answer-option expansion to 10 was partially automated; distractor quality may vary. ## Intended use & out-of-scope - **Intended:** LLM evaluation, leaderboard benchmarking, few-shot and chain-of-thought reasoning assessment, comparative model analysis. - **Out-of-scope:** Training or fine-tuning models you plan to submit to MMLU-Pro leaderboards (contamination); not a comprehensive measure of real-world task performance or non-English capability. _Federated dataset: 2 parquet shards, 4.0 MB total. Queries and downloads stream through the DataBazaar API._ Original supplier listing: MMLU-Pro: Robust Multi-Task LLM Benchmark (12K Questions) 12K challenging multi-discipline multiple-choice questions for benchmarking LLM reasoning. MIT-licensed, parquet format, widely used for model evaluation and leaderboards.