KILT Knowledge-Intensive Language Tasks Benchmark

## Overview KILT (Knowledge-Intensive Language Tasks) is a unified benchmark from Facebook AI Research consolidating 11 datasets across 5 task families (fact-checking, entity linking, slot filling, open-domain QA, dialog generation) all grounded in a single pre-processed Wikipedia dump. The dataset contains between 1M and 10M examples in parquet format, English-only. Original release accompanies the EMNLP 2020 paper (arXiv:2009.02252); the HF mirror was last updated January 2024. ## Schema Schema varies per subtask config, but core fields shared across configs include: - `id` — string — unique example identifier - `input` — string — the query, claim, or dialog context - `output` — list of structs — gold answers/labels, each with: - `answer` — string — gold answer text (for QA/slot filling) or label (for fact-checking) - `provenance` — list of structs — supporting Wikipedia evidence with `wikipedia_id`, `title`, `section`, `start_paragraph_id`, `end_paragraph_id`, `start_character`, `end_character`, `bleu_score`, `meta` - `meta` — struct — task-specific metadata (e.g. left/right context for entity linking, template for slot filling) Subtask configs include: `aidayago2`, `fever`, `hotpotqa`, `nq`, `trex`, `triviaqa_support_only`, `structured_zeroshot`, `wned`, `cweb`, `wow`. ## Sources - HuggingFace: https://huggingface.co/datasets/facebook/kilt_tasks — MIT license - Original paper: Petroni et al., "KILT: a Benchmark for Knowledge Intensive Language Tasks", NAACL 2021 (arXiv:2009.02252) - Source datasets (extended): Natural Questions, AIDA-YAGO, FEVER, HotpotQA, T-REx, TriviaQA, Wizard of Wikipedia, WNED-CWEB, WNED-WIKI, Zero-Shot RE ## Methodology KILT's authors took 11 pre-existing English NLP datasets and re-grounded all of them against a single 2019 Wikipedia snapshot, mapping each example's supporting evidence to specific Wikipedia pages, paragraphs, and character spans. This normalization allows the same retriever and reader to be evaluated across all tasks under identical knowledge-source assumptions, and supports multi-task and transfer learning. Annotations are a mix of crowdsourced, found, and machine-generated depending on the upstream source dataset. ## Known gaps & limitations - English-only (monolingual); not suitable for multilingual evaluation. - Grounded on a 2019 Wikipedia snapshot — temporal drift for any question whose answer has changed since. - Upstream datasets have heterogeneous annotation quality; e.g. TriviaQA is distantly supervised, Wizard of Wikipedia is dialog with crowdsourced grounding. - The `triviaqa_support_only` config is a subset (only examples with KILT-mapped provenance), not full TriviaQA. - Heavy overlap with widely-used eval suites (NQ, HotpotQA, TriviaQA, FEVER) — significant leakage risk if used as training data for models later benchmarked on these. - Provenance spans are mapped automatically and may have alignment errors at character-offset granularity. ## Intended use & out-of-scope - IS for: benchmarking retrieval-augmented generation (RAG) and knowledge-intensive LM systems, multi-task evaluation across QA / fact-checking / entity linking / slot filling / dialog, training retrievers on a unified knowledge source. - NOT for: training models that will later be evaluated on NQ/HotpotQA/TriviaQA/FEVER (leakage); multilingual or non-English evaluation; tasks requiring post-2019 world knowledge. _Federated dataset: 34 parquet shards, 1001.6 MB total. Queries and downloads stream through the DataBazaar API._ Original supplier listing: KILT: Knowledge-Intensive Language Tasks Benchmark Facebook AI's KILT benchmark — 11 datasets across fact-checking, entity linking, slot filling, open-domain QA, and dialog generation, all grounded in a unified Wikipedia snapshot. MIT licensed, parquet format, 1M-10M examples.