BEIR MS MARCO Passage Retrieval Benchmark

## Overview The MS MARCO subset of the BEIR (Benchmarking IR) benchmark — a large-scale English passage retrieval dataset widely used to evaluate dense retrievers, sparse retrievers, and RAG systems. Contains approximately 8.8M passages in the corpus and ~500K natural-language queries derived from anonymized Bing search logs, distributed in Parquet using the standard BEIR retrieval layout. English only. ## Schema - `_id` — string — unique identifier for a document or query - `title` — string — title field (often empty for MS MARCO passages) - `text` — string — passage text (corpus) or query text (queries) The dataset is split into two configurations: `corpus` (one row per passage) and `queries` (one row per query). Relevance judgments (qrels) are distributed via the companion BeIR/msmarco-qrels dataset. ## Sources - BeIR/msmarco on HuggingFace — https://huggingface.co/datasets/BeIR/msmarco — license: CC-BY-SA-4.0 - Original MS MARCO dataset (Microsoft) — https://microsoft.github.io/msmarco/ - BEIR paper (Thakur et al., NeurIPS 2021) — https://arxiv.org/abs/2104.08663 ## Methodology MS MARCO was originally constructed by Microsoft from anonymized Bing query logs, with human annotators identifying relevant passages from web documents that answered each query. BEIR repackages MS MARCO into a uniform JSON/Parquet retrieval format (corpus, queries, qrels) to enable apples-to-apples comparison across its 18 IR tasks. No additional content filtering or augmentation is applied by BEIR beyond schema normalization. ## Known gaps & limitations - English-only; not suitable for multilingual retrieval evaluation. - Queries reflect Bing user search patterns circa 2018, which skews toward informational web search and may underrepresent conversational, multi-hop, or domain-specialist queries. - Many passages lack titles (`title` field often empty). - Sparse relevance judgments: most queries have only 1 judged relevant passage, leading to known underestimation of true recall. - Original Bing log collection date is not precisely documented; treat the dataset as a static 2018-era snapshot. ## Intended use & out-of-scope - Intended for: training and evaluating dense/sparse retrievers, reranker fine-tuning, RAG component benchmarking, zero-shot retrieval research, and as a base corpus for synthetic query generation. - Out-of-scope: training models that will be evaluated on BEIR itself (severe leakage — MS MARCO is part of BEIR), production search over current web content (data is stale), and any non-English IR use case. _Federated dataset: 2 parquet shards, 1.53 GB total. Queries and downloads stream through the DataBazaar API._ Original supplier listing: BEIR: MS MARCO Passage Retrieval Benchmark MS MARCO subset of the BEIR heterogeneous IR benchmark — 8.8M passages and ~500K queries in standard retrieval layout. Parquet format, CC-BY-SA-4.0, English.