OpenOrca Augmented FLAN Instruction Dataset

## Overview OpenOrca is a large-scale instruction-tuning dataset consisting of approximately 4.2M augmented examples drawn from the FLAN Collection, with responses generated by GPT-4 (~1M rows) and GPT-3.5 (~3.2M rows). The data is structured as system-prompt / question / response triples and is designed to replicate, as closely as possible, the dataset distribution described in Microsoft Research's Orca paper (arXiv:2306.02707). Format is Parquet; English-language; size category 1M–10M rows. ## Schema - id — string — unique identifier, prefixed by source FLAN subset (e.g., `cot.`, `niv.`, `flan.`, `t0.`) - system_prompt — string — the system instruction given to the teacher model - question — string — the user prompt / task input drawn from FLAN - response — string — the teacher model (GPT-4 or GPT-3.5) generated answer ## Sources - Open-Orca/OpenOrca on Hugging Face — https://huggingface.co/datasets/Open-Orca/OpenOrca — license: MIT - Underlying prompts derived from the FLAN Collection (arXiv:2301.13688) - Methodology described in the Orca paper (arXiv:2306.02707) ## Methodology The authors sampled prompts from the FLAN Collection following the task mixture and sampling ratios described in the Orca paper. Each prompt was paired with a system message designed to elicit step-by-step / explanation-rich responses, then submitted to GPT-4 and GPT-3.5 to generate teacher responses. The resulting completions were stored alongside the original prompt and system message. No additional human filtering or RLHF was applied; quality depends on the underlying teacher model outputs. ## Known gaps & limitations - English-only; multilingual coverage is not provided. - Responses inherit any biases, hallucinations, or refusals present in GPT-4 / GPT-3.5 from 2023. - The dataset is not deduplicated against common LLM evaluation benchmarks (MMLU, BBH, AGIEval, etc.) — significant leakage risk if used to train models that will be evaluated on FLAN-derived benchmarks. - Data reflects a 2023 snapshot of teacher models and is not updated. - Source does not document a formal quality audit; buyers should validate empirically for their use case. ## Intended use & out-of-scope - Intended: instruction fine-tuning of open LLMs, distillation experiments, research on teacher-student training, prompt-engineering studies. - Out-of-scope: training models for benchmarks derived from FLAN tasks (leakage), production use without bias / safety auditing, non-English applications. _Federated dataset: 2 parquet shards, 2.67 GB total. Queries and downloads stream through the DataBazaar API._ Original supplier listing: OpenOrca — Augmented FLAN Instruction Dataset ~4M GPT-4/GPT-3.5 augmented FLAN instruction-response pairs aligned with the Orca paper distribution. Widely used for instruction tuning and fine-tuning open LLMs.