MathVista Visual Mathematical Reasoning Benchmark

## Overview MathVista is a consolidated benchmark for evaluating mathematical reasoning in visual contexts. It contains roughly 6,141 examples drawn from 28 existing multimodal datasets plus three newly created sets (IQTest, FunctionQA, PaperQA). Each example pairs an image with a math/reasoning question and an answer (multiple-choice or free-form numeric/text). Format is Parquet with image + text modalities. Primarily English with some Chinese and Persian content. ## Schema - `pid` — string — unique problem ID - `question` — string — the natural-language question - `image` — image — the visual context (chart, diagram, figure, table, etc.) - `decoded_image` — image — decoded PIL image - `choices` — list[string] — answer options for multiple-choice items (null otherwise) - `answer` — string — gold answer - `question_type` — string — `multi_choice` or `free_form` - `answer_type` — string — `integer`, `float`, `text`, or `list` - `metadata` — struct — category, task, context, grade, skills, source dataset, language - `query` — string — fully formatted prompt ready for model input - `precision` — float — required numeric precision for free-form answers ## Sources - HuggingFace: https://huggingface.co/datasets/AI4Math/MathVista — License: CC-BY-SA-4.0 - Paper: Lu et al., "MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts", ICLR 2024 (arXiv:2310.02255) ## Methodology Authors aggregated 28 existing multimodal QA / math / science datasets and supplemented them with three new expert-authored sets: IQTest (puzzle-style reasoning), FunctionQA (function plots with algebraic queries), and PaperQA (figures extracted from arXiv papers). Each item was normalized into a unified schema with question, image, answer, and rich metadata tagging the task type, required skills, grade level, and source. A `testmini` split of 1,000 examples is provided for cheap iteration, with the full ~6K `test` split for headline numbers. ## Known gaps & limitations - Test-set answers for the full `test` split are withheld; evaluation requires the leaderboard or GPT-based scoring scripts. - Image quality and resolution vary widely because items are sourced from heterogeneous upstream datasets. - Coverage is dominated by English; Chinese and Persian items are a small minority and not balanced across tasks. - As a widely-used public benchmark, leakage into frontier model training data is a known concern — treat reported scores accordingly. - Some upstream datasets carry their own licenses; CC-BY-SA-4.0 applies to the MathVista aggregation but downstream redistribution of individual images should respect original sources. ## Intended use & out-of-scope - IS for: evaluating vision-language models on mathematical and visual reasoning; designing few-shot prompts; ablation of multimodal reasoning skills; comparison against the public leaderboard. - NOT for: training data for models that will be evaluated on MathVista (contamination risk); general-purpose math tutoring datasets (it is an eval, not an instruction set); production OCR or chart-extraction systems. _Federated dataset: 3 parquet shards, 844.8 MB total. Queries and downloads stream through the DataBazaar API._ Original supplier listing: MathVista — Visual Mathematical Reasoning Benchmark Multimodal math reasoning benchmark with ~6K image+text QA examples across geometry, charts, figures, and scientific diagrams. Standard eval for vision-language models.