Llama-Nemotron Vision-Language Model Dataset

## Overview The Llama-Nemotron VLM Dataset v1 is NVIDIA's open multimodal training corpus released in August 2025, containing between 1M and 10M samples spanning visual question answering, OCR, image captioning, and image-to-text tasks. The dataset was used to train NVIDIA's Llama-Nemotron family of vision-language models and is distributed in JSON format with associated image references. It supports loading via the standard HuggingFace `datasets` library, `pandas`, `polars`, `mlcroissant`, and NVIDIA's Megatron Energon framework. ## Schema The dataset organizes samples across multiple VLM task subsets (e.g., ocr_1 through ocr_9, VQA splits, captioning). Typical fields per sample include: - `image` — string/path — reference to the associated image asset - `question` / `prompt` — string — input query or instruction - `answer` / `response` — string — target output text - `task_type` — string — category label (VQA, OCR, captioning, etc.) - `source` — string — upstream dataset attribution where applicable - Additional metadata fields vary by subset; see HF dataset card for per-subset schema ## Sources - HuggingFace: https://huggingface.co/datasets/nvidia/Llama-Nemotron-VLM-Dataset-v1 — license: CC-BY-4.0 - Related papers: arXiv:2501.14818, arXiv:2502.04223 ## Methodology NVIDIA assembled this dataset by aggregating and curating samples across multiple vision-language task families (VQA, OCR, captioning, image-to-text) to support training the Llama-Nemotron VLM series. Some subsets reference upstream public datasets with task-specific reformatting into a unified instruction-response structure suitable for VLM SFT. The OCR subsets in particular were curated to span diverse document and scene-text scenarios. Refer to the linked arXiv papers for full curation methodology. ## Known gaps & limitations Early releases had a bug where ocr_1 and ocr_3 images were swapped (fixed 2025-08-18), and ocr_9 instructions were revised on 2025-08-19 — users of pre-fix snapshots should refresh. Some subsets reference external image assets that must be downloaded separately rather than being bundled inline. Language coverage and demographic distribution of imagery are not explicitly documented by the source; buyers should validate empirically for fairness-sensitive applications. Dataset has not been deduplicated against common VLM evaluation benchmarks. ## Intended use & out-of-scope - **Intended**: Supervised fine-tuning of vision-language models, multimodal instruction tuning, OCR/VQA capability training, multimodal RAG corpus construction. - **Out-of-scope**: Direct benchmark evaluation without leakage analysis (overlap with MMMU/DocVQA/TextVQA etc. is not characterized); production OCR on sensitive documents without human review. _Federated dataset: 25 parquet shards, 1.60 GB total. Queries and downloads stream through the DataBazaar API._ _PII signals: us_phone×10 present in the sample. Common in public datasets (papers, logs) but worth knowing before joining with private data._ Original supplier listing: Llama-Nemotron VLM Dataset v1 (NVIDIA) NVIDIA's 1M+ sample multimodal dataset for vision-language model training, covering VQA, OCR, captioning, and image-to-text tasks. CC-BY-4.0 licensed.