Beyond the Last Frame: Process-aware Evaluation for Generative Video Reasoning
This work addresses the evaluation gap for generative video reasoning, which is crucial for advancing AI in video-based tasks, though it is incremental as it focuses on benchmarking rather than new model development.
The paper tackles the problem of evaluating generative video reasoning models by proposing a process-aware evaluation paradigm to address outcome-hacking, where models reach correct conclusions through erroneous processes, and finds that state-of-the-art models achieve only about 20% on the new metric.
Recent breakthroughs in video generation have demonstrated an emerging capability termed Chain-of-Frames (CoF) reasoning, where models resolve complex tasks through the generation of continuous frames. While these models show promise for Generative Video Reasoning (GVR), existing evaluation frameworks often rely on single-frame assessments, which can lead to outcome-hacking, where a model reaches a correct conclusion through an erroneous process. To address this, we propose a process-aware evaluation paradigm. We introduce VIPER, a comprehensive benchmark spanning 16 tasks across temporal, structural, symbolic, spatial, physics, and planning reasoning. Furthermore, we propose Process-outcome Consistency (POC@r), a new metric that utilizes VLM-as-Judge with a hierarchical rubric to evaluate both the validity of the intermediate steps and the final result. Our experiments reveal that state-of-the-art video models achieve POC@1.0 only about 20% and exhibit a significant outcome-hacking. We further explore the impact of test-time scaling and sampling robustness, highlighting a substantial gap between current video generation and true generalized visual reasoning. Our benchmark are released at https://github.com/RUCAIBox/VIPER.