textai4bharat/sangrahaindicmultilingualpretrainingllmsouth-asianparquetcc-byai4bharattext-generationlow-resource-languages

Sangraha Indic Language Pretraining Corpus

Name: Sangraha Indic Language Pretraining Corpus
License: https://huggingface.co/datasets/ai4bharat/sangraha

Free

Open dataset

Sample structure: 100 / 100

2 download links issued

Seller: DataBazaar

Already have an account? Log in

Agent? Connect your account →

Source, license and coverage

Supplier documentation. These claims are separate from the automated sample score. A listing edit date is not a data freshness date.

License: cc-by-4.0
Source / creator: ai4bharat/sangraha
Collection method: Per the authors' paper, Sangraha is built from three streams: (1) **Verified** — text scraped from manually verified high-quality Indic websites; (2) **Unverified** — text from broader web sources passed through perplexity-based and heuristic filters; (3) **Synthetic** — large-scale translations of English content into Indic languages using NMT systems. All streams undergo language identification, deduplication, toxicity/PII filtering, and quality scoring via the open-source `setu` cleaning pipeline.
Coverage start: Not documented
Coverage end: Not documented
Data last updated: Not documented
Update schedule: Not documented

Source documentation ↗

License terms ↗

- Coverage is uneven across the 22 languages — high-resource languages (Hindi, Bengali, Tamil) dominate token counts; low-resource languages (Kashmiri, Sindhi, Sanskrit) have significantly less. - Synthetic (translated) portions inherit NMT artifacts and may not reflect natural native usage. - Deduplication is internal; no guarantee of disjointness from common Indic eval benchmarks — leakage risk exists. - PII and toxicity filters are heuristic; residual unsafe content possible. - Web-sourced material reflects internet demographic biases for each language community.

Sample structure score: 100 / 100

This automated check describes the inspected sample, not factual accuracy, legal rights, representativeness, or the quality of the entire dataset. It is not a customer rating.

Assessed 10 sample records (JSON) on 2026-10-09. All records in the provided sample were checked.

Check	Points	Evidence
Populated cells	50 / 50	30 of 30 top-level cells contain a value. Null, absent and blank values count as missing; zero and false count as populated.
Consistent value types	30 / 30	30 of 30 populated cells match their column's most common observed type. Types are inferred, not checked against real-world truth.
Consistent record shape	20 / 20	10 of 10 records have the expected fields. CSV/TSV use the header width; JSON uses the union of observed keys.

Field-level findings and improvements

Check missing cells and mixed types below. Document intentional missing values or mixed types in your field descriptions. Do not fill legitimate unknowns with invented values just to increase this score.

Field	Missing cells	Most common type	Other populated types
doc_id	0 / 10	string	0 / 10
type	0 / 10	string	0 / 10
text	0 / 10	string	0 / 10

How the score is calculated, its limitations, and how to correct an assessment →

About this data

Indic-language pretraining text across 22 languages, combining web sources and machine-translated corpora. Translated portions may contain synthetic artifacts, and coverage is uneven across languages.

Retrieve with your agent or Python

Create an account and configure DATABAZAAR_API_KEY. This example retrieves free or already purchased data; it never makes a purchase. For a multi-file dataset, choose a file index from the manifest.

Download the Python example

python3 retrieve-dataset.py 1275f2a9-9b18-475d-8466-e45784041829 --output dataset.bin

Full supplier documentation

## Overview Sangraha is the largest high-quality, cleaned Indic-language pretraining corpus, containing approximately 251 billion tokens across 22 Indian languages. It is distributed as parquet files via Hugging Face and is suitable for pretraining and continued pretraining of large language models targeting South Asian languages. Content was assembled by AI4Bharat (IIT Madras) from curated web sources, existing multilingual corpora, and large-scale translation pipelines, then filtered through a multi-stage cleaning process. ## Schema - `text` — string — raw text document content - `language` — string — ISO language code (one of 22 Indic languages + English) - `source` — string — origin partition (e.g., verified, unverified, synthetic) - `doc_id` — string — document identifier - Additional metadata columns vary by subset (verified vs synthetic vs unverified splits) ## Sources - Hugging Face: https://huggingface.co/datasets/ai4bharat/sangraha — license: CC-BY-4.0 - Paper: https://arxiv.org/abs/2403.06350 (Khan et al., "IndicLLMSuite") - Curation pipelines: AI4Bharat GitHub (setu pipeline) ## Methodology Per the authors' paper, Sangraha is built from three streams: (1) **Verified** — text scraped from manually verified high-quality Indic websites; (2) **Unverified** — text from broader web sources passed through perplexity-based and heuristic filters; (3) **Synthetic** — large-scale translations of English content into Indic languages using NMT systems. All streams undergo language identification, deduplication, toxicity/PII filtering, and quality scoring via the open-source `setu` cleaning pipeline. ## Known gaps & limitations - Coverage is uneven across the 22 languages — high-resource languages (Hindi, Bengali, Tamil) dominate token counts; low-resource languages (Kashmiri, Sindhi, Sanskrit) have significantly less. - Synthetic (translated) portions inherit NMT artifacts and may not reflect natural native usage. - Deduplication is internal; no guarantee of disjointness from common Indic eval benchmarks — leakage risk exists. - PII and toxicity filters are heuristic; residual unsafe content possible. - Web-sourced material reflects internet demographic biases for each language community. ## Intended use & out-of-scope - **Intended:** pretraining or continued pretraining of multilingual / Indic LLMs, tokenizer training, language-specific RAG corpora, linguistic research on Indian languages. - **Out-of-scope:** evaluation benchmarks (data may overlap with public Indic evals), production deployment without additional safety filtering, or use cases requiring guaranteed human-authored (non-synthetic) text without filtering by the `source` field. _Federated dataset: 1,950 parquet shards, 431.28 GB total. Queries and downloads stream through the DataBazaar API._ Original supplier listing: Sangraha — 251B Token Indic Language Pretraining Corpus (22 Languages) Largest cleaned Indic-language pretraining dataset: 251B tokens across 22 Indian languages, curated from web sources, multilingual corpora, and large-scale translations. CC-BY-4.0, parquet format.

Schema

Name	Type	Description
doc_id	VARCHAR	Unique SHA-1 hash identifier for each document in the corpus.
type	VARCHAR	Content source type: 'web', 'synthetic', or other origin category.
text	VARCHAR	Raw text document content in Indic languages or English, may contain multiple languages within single document.

Sample Data

Preview a sample of the data before downloading.

Public sample only. Sign in to retrieve the full dataset, including free datasets.

For AI Agents

Via MCP Server

# 1. Add to your agent's MCP config (claude_desktop_config.json or similar):
{
  "mcpServers": {
    "databazaar": { "command": "npx", "args": ["databazaar-mcp"] }
  }
}

# 2. Your agent can then call:
search_datasets({ query: "Sangraha Indic Language Pretra" })
// Found: 1275f2a9-9b18-475d-8466-e45784041829
get_download_url({ dataset_id: "1275f2a9-9b18-475d-8466-e45784041829" })  // free — sign in with MCP OAuth first

Via REST API

# Free dataset — sign in or use your account API key:
curl https://api.databazaar.io/datasets/1275f2a9-9b18-475d-8466-e45784041829/download-url -H "Authorization: Bearer $DATABAZAAR_API_KEY"