UltraFeedback Preference Dataset

## Overview UltraFeedback is a large-scale, fine-grained preference dataset built for training reward models, critic models, and RLHF/DPO pipelines. It contains ~64k prompts sourced from diverse instruction datasets, each answered by 4 different LLMs, yielding ~256k response samples. Each response is annotated by GPT-4 across multiple fine-grained aspects (instruction-following, truthfulness, honesty, helpfulness) with both numeric scores and textual critiques. Format is JSON; English-language; ~10K–100K row-group size category on the Hub. Released by OpenBMB (Tsinghua); paper arXiv:2310.01377. ## Schema - `source` — string — origin of the prompt (UltraChat, ShareGPT, Evol-Instruct, TruthfulQA, FalseQA, FLAN) - `instruction` — string — the user prompt - `models` — list[string] — names of the 4 LLMs that produced completions - `completions` — list[object] — per-response records, each containing: - `model` — string — responding model id - `response` — string — generated text - `annotations` — object — GPT-4 fine-grained ratings on instruction-following, truthfulness, honesty, helpfulness, each with rating + rationale - `overall_score` — float — aggregate quality score - `correct_answers` / `incorrect_answers` — list[string] — present for TruthfulQA/FalseQA-derived prompts ## Sources - HuggingFace: https://huggingface.co/datasets/openbmb/UltraFeedback — License: MIT - Paper: https://arxiv.org/abs/2310.01377 - GitHub: https://github.com/OpenBMB/UltraFeedback ## Methodology Prompts were sampled from six instruction corpora (UltraChat, ShareGPT, Evol-Instruct, TruthfulQA, FalseQA, FLAN) to maximize task and style diversity. For each prompt, 4 responses were generated by a pool of ~17 LLMs spanning GPT-4, GPT-3.5, Bard, LLaMA-2 variants, Falcon, MPT, StarChat, UltraLM, Vicuna, WizardLM, and others, with model selection randomized to balance coverage. GPT-4 was then used as the annotator to produce per-aspect ratings and natural-language critiques along four axes plus an overall score, following a structured rubric defined in the paper. ## Known gaps & limitations - All preference labels are GPT-4-generated, not human; inherits GPT-4's biases (verbosity preference, stylistic preferences, known calibration issues on safety/truthfulness edge cases). - English-only. - Responding-model pool reflects late-2023 LLMs; newer-generation completions are absent. - Source prompts include TruthfulQA and FLAN content — leakage risk if used to train models that will later be evaluated on those benchmarks. - A known annotation bug in the original release was patched in a later revision; downstream cleaned variants (e.g. ultrafeedback_binarized) exist and may be preferable for some uses. ## Intended use & out-of-scope - IS for: training reward models, DPO/IPO/KTO preference fine-tuning, critic/judge model training, RLHF research, studying GPT-4-as-judge behavior. - NOT for: ground-truth human preference modeling (labels are synthetic); benchmark training where TruthfulQA/FLAN leakage matters; non-English alignment work. _Federated dataset: 2 parquet shards, 307.9 MB total. Queries and downloads stream through the DataBazaar API._ Original supplier listing: UltraFeedback — Large-Scale Fine-Grained Preference Dataset 64k prompts × 4 LLM responses (256k samples) with fine-grained GPT-4 preference annotations across instruction-following, truthfulness, honesty, and helpfulness. Canonical dataset for reward model and RLHF training.