ShotBench Cinematic Understanding Benchmark

## Overview ShotBench is the official test set for evaluating expert-level cinematic understanding in vision-language models. It contains 3,572 question-answer pairs distributed across 3,049 images and 464 video clips, primarily sourced from films nominated for the Academy Award for Best Cinematography. Format is CSV with paired image/video media. Released by Vchitect in 2025. ## Schema - question — string — multiple-choice question about a cinematic dimension (shot size, framing, camera angle, lens, lighting, composition, movement) - options — string — answer choices - answer — string — ground-truth answer label - media_path — string — pointer to associated image or video clip - media_type — string — image or video - dimension/category — string — which cinematic skill is being probed - source_film — string — originating film reference (where provided) ## Sources - Vchitect/ShotBench on Hugging Face — https://huggingface.co/datasets/Vchitect/ShotBench — Apache-2.0 - Paper: ShotBench (arXiv:2506.21356) ## Methodology The authors curated stills and clips from Oscar-nominated cinematography films to ensure high visual quality and strong cinematic style. Domain experts annotated each sample across cinematic dimensions (shot size, framing, camera angle, lens type, lighting type/condition, composition, camera movement), producing multiple-choice QA pairs. The set is intended as a held-out test benchmark rather than a training corpus. ## Known gaps & limitations The benchmark is English-only and skews toward award-nominated Western cinema, which may not represent global filmmaking traditions, TV, documentary, or amateur footage. With 3,572 items, statistical resolution between closely-matched models may be limited. As a public benchmark, leakage risk exists if used in training. Source does not exhaustively document annotator agreement; buyers should validate empirically. ## Intended use & out-of-scope - Intended: evaluating vision-language and video-language models on cinematography understanding; comparative model benchmarking; probing VLM perception of camera/lens/lighting concepts. - Out-of-scope: training data for production models (leakage risk); general video understanding outside cinematic style; non-English deployment. Original supplier listing: ShotBench: Cinematic Understanding Benchmark for VLMs 3,572 expert-level QA pairs over 3,049 images and 464 video clips from Oscar-nominated cinematography films, for evaluating cinematic understanding in vision-language models.