MMLU Benchmark

## Overview MMLU (Measuring Massive Multitask Language Understanding) is a multiple-choice question-answering benchmark covering 57 subjects spanning humanities, social sciences, STEM, and professional domains (law, medicine, ethics, etc.). It is one of the most widely used LLM evaluation suites. The dataset contains ~116K questions in Parquet format, split into auxiliary train, dev (few-shot exemplars), validation, and test sets. English-only, monolingual. ## Schema - question — string — the multiple-choice question stem - choices — list[string] — four answer options (A/B/C/D) - answer — int — index (0-3) of the correct choice - subject — string — one of 57 subject labels (e.g., `high_school_physics`, `professional_law`, `moral_scenarios`) ## Sources - Hugging Face: https://huggingface.co/datasets/cais/mmlu — License: MIT - Original paper: Hendrycks et al., "Measuring Massive Multitask Language Understanding" (ICLR 2021), arXiv:2009.03300 ## Methodology Questions were hand-collected by the authors and collaborators from freely available online sources including practice exams (GRE, USMLE, AP, bar exam prep), textbook exercises, and Oxford University Press materials. Each subject was curated by topic experts to ensure coverage from elementary through professional difficulty. No additional annotation layer was applied; the original answer keys from source materials are preserved. The data is distributed as Parquet with one configuration per subject plus an `all` config. ## Known gaps & limitations MMLU is known to contain label errors and ambiguous questions — follow-up work (MMLU-Redux, MMLU-Pro) has documented and corrected hundreds of issues. The benchmark is heavily contaminated in modern pretraining corpora, so test-set leakage is a serious concern for any model trained on broad web data post-2020. English-only; cultural bias toward US academic and professional curricula. Some subjects (e.g., moral_scenarios) have idiosyncratic formats that produce unstable scores. ## Intended use & out-of-scope - IS for: zero-shot and few-shot LLM evaluation, comparing model knowledge breadth, ablation studies, reporting standard leaderboard numbers. - NOT for: fine-tuning or training (severe leakage risk — will inflate downstream eval scores); not a measure of reasoning depth or factual grounding beyond multiple-choice recall; not suitable as ground truth for non-English models without translation. _Federated dataset: 176 parquet shards, 98.8 MB total. Queries and downloads stream through the DataBazaar API._ Original supplier listing: MMLU — Massive Multitask Language Understanding Benchmark 57-subject multiple-choice benchmark (humanities, STEM, social sciences, professional) for evaluating LLM knowledge and reasoning. ~116K questions across dev/val/test splits. MIT licensed.