EO-Data-1.5M Interleaved Vision-Text-Action Dataset

## Overview EO-Data-1.5M is a large-scale multimodal embodied reasoning dataset containing ~1.5 million interleaved vision-text-action samples designed for training generalist robot policies and vision-language-action (VLA) models. It is the first large-scale interleaved embodied dataset emphasizing temporal dynamics and causal dependencies among vision, language, and action modalities. Stored as parquet with image and text modalities. ## Schema The dataset uses an interleaved multimodal format with the following typical fields (exact column list documented on the HF page): - `images` — image/bytes — frames or observations from robot episodes - `text` — string — interleaved instructions, reasoning traces, captions - `actions` — sequence/array — robot action tokens or low-level controls - `task` / `instruction` — string — natural language task description - `episode_id` — string — episode/segment identifier - `source` — string — originating sub-dataset or robot platform - `modality_order` — sequence — order in which vision/text/action tokens are interleaved - +additional metadata columns (see HF dataset page) ## Sources - IPEC-COMMUNITY/EO-Data1.5M on Hugging Face — https://huggingface.co/datasets/IPEC-COMMUNITY/EO-Data1.5M — Apache-2.0 - Associated paper: arXiv:2508.21112 ## Methodology The dataset was assembled by the IPEC community as a curated, interleaved corpus combining vision frames, natural-language instructions/reasoning, and robot actions across multiple embodied AI and manipulation sources. Samples are formatted in an interleaved sequence format (rather than separate vision/text/action streams) to preserve temporal and causal structure across modalities, making it suitable for autoregressive VLA pretraining. See the linked paper for collection and normalization details. ## Known gaps & limitations Language is English-only. The dataset aggregates multiple upstream robotics sources and inherits their domain biases (manipulation tasks dominate; locomotion and dexterous tasks may be underrepresented). The source does not exhaustively document per-sub-dataset provenance or deduplication against common robot-learning benchmarks; buyers should validate empirically before using in evals. Action spaces vary across constituent sources and may require normalization for downstream policy training. ## Intended use & out-of-scope - Intended: pretraining and fine-tuning of vision-language-action models, embodied reasoning research, multimodal robot policy learning, interleaved-modality pretraining. - Out-of-scope: deployment on physical robots without sim-to-real validation; benchmark training without checking for leakage against standard robot-learning evals (e.g., LIBERO, RoboCasa). _Federated dataset: 1,235 parquet shards, 172.84 GB total. Queries and downloads stream through the DataBazaar API._ Original supplier listing: EO-Data-1.5M: Interleaved Vision-Text-Action Dataset for Embodied AI 1.5M-sample interleaved vision-language-action dataset for embodied AI and robot learning, emphasizing temporal dynamics and causal dependencies across modalities. Apache-2.0 licensed, parquet format.