textSalesforce/wikitextlanguage-modelingwikipediabenchmarktextpretrainingenglishperplexitynlp

WikiText-2 & WikiText-103 Language Modeling Corpora

Free

Open dataset

Sample structure: 80 / 100

2 download links issued

Seller: DataBazaar

Already have an account? Log in

Agent? Connect your account →

Source, license and coverage

Supplier documentation. These claims are separate from the automated sample score. A listing edit date is not a data freshness date.

License: cc-by-sa-3.0,gfdl
Source / creator: Salesforce/wikitext
Collection method: The authors extracted full article text from the set of Wikipedia articles tagged as "Good" or "Featured" — a quality-curated subset of English Wikipedia. Compared to PTB, case, punctuation, and numbers are preserved. The `-raw` variants keep the original text; the non-raw variants apply Moses tokenizer-style preprocessing and replace rare words (frequency < 3) with `<unk>`. Articles are concatenated with section headings preserved, and the corpus is split into train/validation/test partitions at the article level.
Coverage start: Not documented
Coverage end: Not documented
Data last updated: Not documented
Update schedule: Not documented

Source documentation ↗

- English-only, monolingual; not suitable for multilingual evaluation. - Source snapshot is from ~2016; content reflects Wikipedia of that era and does not include newer articles or edits. - Heavy overlap risk with modern LLM pretraining corpora — WikiText is widely used as an eval and many models have effectively seen this data during training. Reported perplexity on WikiText-103 should be interpreted with leakage in mind. - Domain is restricted to high-quality encyclopedic prose; not representative of conversational, code, or noisy web text. - Non-raw variants' `<unk>` tokenization is lossy and not appropriate for modern subword tokenizers.

Sample structure score: 80 / 100

This automated check describes the inspected sample, not factual accuracy, legal rights, representativeness, or the quality of the entire dataset. It is not a customer rating.

Assessed 10 sample records (JSON) on 2026-10-09. All records in the provided sample were checked.

Check	Points	Evidence
Populated cells	30 / 50	6 of 10 top-level cells contain a value. Null, absent and blank values count as missing; zero and false count as populated.
Consistent value types	30 / 30	6 of 6 populated cells match their column's most common observed type. Types are inferred, not checked against real-world truth.
Consistent record shape	20 / 20	10 of 10 records have the expected fields. CSV/TSV use the header width; JSON uses the union of observed keys.

Field-level findings and improvements

Check missing cells and mixed types below. Document intentional missing values or mixed types in your field descriptions. Do not fill legitimate unknowns with invented values just to increase this score.

Field	Missing cells	Most common type	Other populated types
text	4 / 10	string	0 / 6

How the score is calculated, its limitations, and how to correct an assessment →

About this data

Benchmark corpus of verified Good and Featured Wikipedia articles in raw and tokenized variants, used for language modeling evaluation.

Retrieve with your agent or Python

Create an account and configure DATABAZAAR_API_KEY. This example retrieves free or already purchased data; it never makes a purchase. For a multi-file dataset, choose a file index from the manifest.

Download the Python example

python3 retrieve-dataset.py a8972314-0c46-4dfe-9418-401425b1615c --output dataset.bin

Full supplier documentation

## Overview WikiText is a long-standing language modeling benchmark consisting of over 100 million tokens extracted from verified Good and Featured articles on English Wikipedia. The dataset ships in four configurations — `wikitext-2-v1`, `wikitext-2-raw-v1`, `wikitext-103-v1`, and `wikitext-103-raw-v1` — each split into train/validation/test. WikiText-103 contains ~1.8M rows of text; WikiText-2 is ~2x larger than preprocessed PTB and WikiText-103 is ~110x larger. Single text column per row, distributed as Parquet. ## Schema - `text` — string — a line of Wikipedia article text (may be a paragraph, heading like `= Title =`, or empty line preserving article structure) Only one column. Article boundaries are encoded inline via `= Heading =` markers preserved from the source Wikipedia markup. ## Sources - Hugging Face: https://huggingface.co/datasets/Salesforce/wikitext — License: CC-BY-SA-3.0 and GFDL - Original paper: Merity et al., "Pointer Sentinel Mixture Models" (arXiv:1609.07843) ## Methodology The authors extracted full article text from the set of Wikipedia articles tagged as "Good" or "Featured" — a quality-curated subset of English Wikipedia. Compared to PTB, case, punctuation, and numbers are preserved. The `-raw` variants keep the original text; the non-raw variants apply Moses tokenizer-style preprocessing and replace rare words (frequency < 3) with `<unk>`. Articles are concatenated with section headings preserved, and the corpus is split into train/validation/test partitions at the article level. ## Known gaps & limitations - English-only, monolingual; not suitable for multilingual evaluation. - Source snapshot is from ~2016; content reflects Wikipedia of that era and does not include newer articles or edits. - Heavy overlap risk with modern LLM pretraining corpora — WikiText is widely used as an eval and many models have effectively seen this data during training. Reported perplexity on WikiText-103 should be interpreted with leakage in mind. - Domain is restricted to high-quality encyclopedic prose; not representative of conversational, code, or noisy web text. - Non-raw variants' `<unk>` tokenization is lossy and not appropriate for modern subword tokenizers. ## Intended use & out-of-scope - **For:** Language modeling perplexity evaluation, small-scale pretraining experiments, tokenizer benchmarking, long-context evaluation baselines, RAG retrieval-corpus experiments on clean prose. - **Not for:** Fair benchmarking of frontier LLMs (likely in training data — leakage risk); multilingual tasks; up-to-date factual QA; modeling informal or noisy text. _Federated dataset: 14 parquet shards, 613.9 MB total. Queries and downloads stream through the DataBazaar API._ Original supplier listing: WikiText (WikiText-2 & WikiText-103) Language Modeling Corpus Canonical language modeling benchmark: 100M+ tokens from verified Good/Featured Wikipedia articles. Includes WikiText-2 and WikiText-103 in raw and tokenized variants. Parquet format, CC-BY-SA 3.0.

Schema

Name	Type	Description
text	VARCHAR	Line of Wikipedia article text, including paragraphs, headings (formatted as = Title =), and empty lines preserving article structure

Sample Data

Preview a sample of the data before downloading.

Public sample only. Sign in to retrieve the full dataset, including free datasets.

For AI Agents

Via MCP Server

# 1. Add to your agent's MCP config (claude_desktop_config.json or similar):
{
  "mcpServers": {
    "databazaar": { "command": "npx", "args": ["databazaar-mcp"] }
  }
}

# 2. Your agent can then call:
search_datasets({ query: "WikiText-2 & WikiText-103 Lang" })
// Found: a8972314-0c46-4dfe-9418-401425b1615c
get_download_url({ dataset_id: "a8972314-0c46-4dfe-9418-401425b1615c" })  // free — sign in with MCP OAuth first

Via REST API

# Free dataset — sign in or use your account API key:
curl https://api.databazaar.io/datasets/a8972314-0c46-4dfe-9418-401425b1615c/download-url -H "Authorization: Bearer $DATABAZAAR_API_KEY"