A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairs
This addresses the challenge of accurately assessing physical understanding in video-language models for researchers and developers, though it is incremental as it builds on existing benchmark methods.
The paper tackles the problem of inflated performance scores in video question-answering benchmarks due to shortcut solutions, by introducing the Minimal Video Pairs (MVP) benchmark, which requires models to answer correctly on visually similar video pairs with opposing answers, resulting in a best model performance of 40.2% compared to human performance of 92.9%.
Existing benchmarks for assessing the spatio-temporal understanding and reasoning abilities of video language models are susceptible to score inflation due to the presence of shortcut solutions based on superficial visual or textual cues. This paper mitigates the challenges in accurately assessing model performance by introducing the Minimal Video Pairs (MVP) benchmark, a simple shortcut-aware video QA benchmark for assessing the physical understanding of video language models. The benchmark is comprised of 55K high-quality multiple-choice video QA examples focusing on physical world understanding. Examples are curated from nine video data sources, spanning first-person egocentric and exocentric videos, robotic interaction data, and cognitive science intuitive physics benchmarks. To mitigate shortcut solutions that rely on superficial visual or textual cues and biases, each sample in MVP has a minimal-change pair -- a visually similar video accompanied by an identical question but an opposing answer. To answer a question correctly, a model must provide correct answers for both examples in the minimal-change pair; as such, models that solely rely on visual or textual biases would achieve below random performance. Human performance on MVP is 92.9\%, while the best open-source state-of-the-art video-language model achieves 40.2\% compared to random performance at 25\%.