MAPS Multilingual Agentic AI Benchmark

## Overview MAPS (Multilingual Agentic AI Benchmark for Global Agent Performance and Security) is a benchmark of 805 tasks designed to evaluate agentic AI systems across 11 languages and diverse tasks. It combines 405 performance-oriented tasks drawn from GAIA, SWE-bench, and MATH with 400 tasks from the Agent Security Benchmark, enabling systematic analysis of agent behavior under multilingual conditions for both capability and safety dimensions. Text modality, 1K–10K size range. ## Schema Schema follows standard agent-eval task records. Typical fields include: - `task_id` — string — unique identifier for each task - `language` — string — ISO language code (ar, en, ja, es, ko, hi, ru, he, pt, de, it) - `source_dataset` — string — origin benchmark (GAIA, SWE-bench, MATH, Agent Security Benchmark) - `category` — string — performance vs. security classification - `question` / `prompt` — string — the task instruction in the target language - `answer` / `expected_output` — string — ground-truth answer or expected behavior - `metadata` — object — task-specific fields (difficulty, tools required, threat model, etc.) Exact columns vary per source subset; consult the HF dataset viewer for per-split fields. ## Sources - Fujitsu-FRE/MAPS on Hugging Face — https://huggingface.co/datasets/Fujitsu-FRE/MAPS — license: CC-BY-4.0 - Underlying source benchmarks: GAIA (arXiv:2311.12983), SWE-bench (arXiv:2310.06770), MATH (arXiv:2103.03874), Agent Security Benchmark (arXiv:2410.02644) ## Methodology The authors curated 405 performance tasks from established agentic/reasoning benchmarks (GAIA, SWE-bench, MATH) and 400 tasks from the Agent Security Benchmark, then produced multilingual versions across 11 languages to enable cross-lingual evaluation of both capability and security. Translation/localization methodology is described in the accompanying paper; tasks preserve original answer keys for evaluation parity with the English source benchmarks. ## Known gaps & limitations Multilingual benchmarks built via translation can carry translation artifacts and may not capture culturally-specific reasoning. Coverage is limited to 11 languages, weighted toward high-resource languages. Because tasks derive from public benchmarks (GAIA, SWE-bench, MATH), there is meaningful overlap with widely-used training corpora — leakage risk for any model trained on web-scale data. The 805-task scale is modest and may have high evaluation variance. Source does not exhaustively document per-language quality assurance; buyers should validate empirically for their use case. ## Intended use & out-of-scope - IS for: multilingual evaluation of agentic AI systems, cross-lingual performance/security comparisons, regression testing of agent frameworks across languages. - NOT for: training data (high contamination risk against public benchmarks), production safety certification, or as a substitute for native-language adversarial red-teaming. _Federated dataset: 49 parquet shards, 8.9 MB total. Queries and downloads stream through the DataBazaar API._ Original supplier listing: MAPS: Multilingual Agentic AI Benchmark (Performance & Security) 805-task multilingual benchmark for evaluating agentic AI across 11 languages, combining performance tasks (GAIA, SWE-bench, MATH) with Agent Security Benchmark tasks. CC-BY-4.0.