VCRBench: Exploring Long-form Causal Reasoning Capabilities of Large Video Language Models
This addresses the lack of benchmarks for evaluating causal reasoning in video-language models, which is important for researchers in video understanding and multimodal AI, though it is incremental as it builds on existing LVLM frameworks.
The authors tackled the problem of evaluating video-based long-form causal reasoning in Large Video Language Models (LVLMs) by introducing VCRBench, a novel benchmark using procedural videos with shuffled steps, and found that state-of-the-art LVLMs struggle with this task. They proposed a modular Recognition-Reasoning Decomposition (RRD) approach that boosted accuracy by up to 25.2% on VCRBench.
Despite recent advances in video understanding, the capabilities of Large Video Language Models (LVLMs) to perform video-based causal reasoning remains underexplored, largely due to the absence of relevant and dedicated benchmarks for evaluating causal reasoning in visually grounded and goal-driven settings. To fill this gap, we introduce a novel benchmark named Video-based long-form Causal Reasoning (VCRBench). We create VCRBench using procedural videos of simple everyday activities, where the steps are deliberately shuffled with each clip capturing a key causal event, to test whether LVLMs can identify, reason about, and correctly sequence the events needed to accomplish a specific goal. Moreover, the benchmark is carefully designed to prevent LVLMs from exploiting linguistic shortcuts, as seen in multiple-choice or binary QA formats, while also avoiding the challenges associated with evaluating open-ended QA. Our evaluation of state-of-the-art LVLMs on VCRBench suggests that these models struggle with video-based long-form causal reasoning, primarily due to their difficulty in modeling long-range causal dependencies directly from visual observations. As a simple step toward enabling such capabilities, we propose Recognition-Reasoning Decomposition (RRD), a modular approach that breaks video-based causal reasoning into two sub-tasks of video recognition and causal reasoning. Our experiments on VCRBench show that RRD significantly boosts accuracy on VCRBench, with gains of up to 25.2%. Finally, our thorough analysis reveals interesting insights, for instance, that LVLMs primarily rely on language knowledge for complex video-based long-form causal reasoning tasks.