Bhasha SFT Multilingual Instruction-Response Pairs

## Overview Bhasha SFT is a curated collation of open-source Supervised Fine-Tuning (SFT) datasets aggregated by Soket AI Labs for training multilingual Large Language Models. It contains over 13 million instruction-response instances spanning four languages — Hindi, Bengali, Gujarati, and English — combining both human-annotated and synthetically generated examples. Distributed in Parquet format and accessible via HuggingFace datasets, Dask, Polars, and mlcroissant. ## Schema - instruction — string — the prompt or task instruction given to the model - input — string — optional context or input text supporting the instruction - output — string — the target response - language — string — language code (hi, en, gu, bn) - source — string — upstream dataset name this instance was sourced from - (additional metadata columns may vary by source partition) ## Sources - soketlabs/bhasha-sft on HuggingFace: https://huggingface.co/datasets/soketlabs/bhasha-sft — licenses: cc-by-4.0, apache-2.0, mit (inherited from constituent datasets) ## Methodology The dataset is a collation of multiple existing open-source instruction-tuning datasets. Soket AI Labs aggregated public SFT corpora covering Indic languages plus English, normalized them into a common instruction/input/output schema, and stored them in Parquet. The mixture includes both human-annotated data (e.g. translated/native instruction sets) and synthetic data generated via LLMs. Specific upstream sources are tracked per-row via the source field. ## Known gaps & limitations The dataset card does not provide a detailed per-source breakdown or deduplication report. Synthetic portions inherit any biases or hallucinations from the generator models. Coverage across the three Indic languages is likely uneven (Hindi typically dominates Indic SFT collations). Quality varies by upstream source — some constituent datasets are machine-translated and may contain translation artifacts. License is a mix (cc-by-4.0, apache-2.0, mit) depending on the original source, so downstream users should track source attribution per-row for compliance. ## Intended use & out-of-scope - IS for: supervised fine-tuning and instruction-tuning of multilingual / Indic LLMs, RAG eval sets for Indian languages, distillation data, multilingual instruction-following research. - NOT for: drop-in production training without dedup/quality filtering; not deduplicated against common eval benchmarks (MMLU, IndicXTREME, etc.) — leakage risk; not a substitute for native-quality human-annotated data in low-resource Indic languages. _Federated dataset: 61 parquet shards, 9.24 GB total. Queries and downloads stream through the DataBazaar API._ Original supplier listing: Bhasha SFT — 13M+ Multilingual Instruction-Response Pairs (Hindi, Bengali, Gujarati, English) 13M+ instruction-response pairs across Hindi, Bengali, Gujarati, and English for supervised fine-tuning of multilingual LLMs. Mix of human-annotated and synthetic data from open-source SFT collections, curated by Soket AI Labs.