Aya Collection Multilingual Instruction Instances

## Overview The Aya Collection is a massive multilingual instruction-tuning dataset consisting of ~513 million instances across 115+ languages, assembled by Cohere Labs (formerly Cohere For AI). This particular distribution is a re-upload of the original `CohereForAI/aya_collection` re-sharded by language rather than by source dataset, so downstream users interested in a single language (or a small subset) can pull just the relevant shards. Data is stored as parquet, tabular text modality, and is directly loadable via `datasets`, `dask`, `polars`, or `mlcroissant`. ## Schema - `id` — int — unique instance ID - `inputs` — string — instruction / prompt text in the target language - `targets` — string — expected response/completion - `dataset_name` — string — source sub-dataset within the Aya Collection (e.g. templated NLP task, translated instruction set, human-annotated Aya Dataset entries) - `sub_dataset_name` — string — finer-grained source identifier - `task_type` — string — task category (translation, QA, summarization, classification, generation, etc.) - `template_id` — int — template index used when the row was templated from a base NLP dataset - `language` — string — ISO 639-3 language code (matches the shard) - `script` — string — writing script code - `split` — string — train/validation/test partition as inherited from source ## Sources - Hugging Face: https://huggingface.co/datasets/CohereLabs/aya_collection_language_split — license: Apache-2.0 - Paper: Singh et al., "Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning", arXiv:2402.06619 - Upstream: `CohereForAI/aya_collection` (same data, folder-by-dataset layout) ## Methodology The Aya Collection was assembled by Cohere Labs through three pipelines: (1) templating — applying human-curated instruction templates to dozens of existing multilingual NLP datasets to produce instruction/response pairs; (2) translating — machine-translating widely used English instruction datasets into many languages; and (3) the human-annotated Aya Dataset contributed by a global community of native-speaker volunteers. The language-split version re-organizes the same underlying rows into per-language parquet shards with no content modification. ## Known gaps & limitations Coverage and quality vary substantially by language: high-resource languages (eng, spa, fra, zho, etc.) have orders of magnitude more instances than low-resource ones (e.g. ace, bbc, bjn, taq, nso). A large fraction of non-English data originates from machine translation and templating, so fluency and cultural appropriateness are uneven; only a minority of rows are natively human-authored. Templated rows can be repetitive across `template_id`. The collection overlaps with many public NLP benchmarks, so it is NOT deduplicated against common eval suites — using it for SFT introduces meaningful benchmark-contamination risk. The Cohere team documents these tradeoffs in the paper; buyers should validate per-language quality empirically before training. ## Intended use & out-of-scope - IS for: multilingual instruction-tuning and SFT, multilingual RAG corpora, cross-lingual transfer research, language-specific fine-tuning where pulling a single-language shard is easier than filtering the full dump. - NOT for: clean held-out evaluation of multilingual benchmarks (leakage risk via templating sources), nor as ground truth for low-resource translation quality (much of the low-resource data is itself MT-generated). _Federated dataset: 491 parquet shards, 144.55 GB total. Queries and downloads stream through the DataBazaar API._ Original supplier listing: Aya Collection (Language-Split) — 513M Multilingual Instruction Instances Cohere Labs' Aya Collection re-uploaded with per-language splits. 513M multilingual instruction-tuning instances across 115+ languages in parquet. Apache-2.0.