GraphThinker: Reinforcing Video Reasoning with Event Graph Thinking
This addresses the issue of implicit causal relationships in videos for AI systems, but it is incremental as it builds on existing multimodal large language models.
The paper tackles the problem of hallucinations in video reasoning by proposing GraphThinker, a method that constructs event-based scene graphs and uses reinforcement finetuning with visual attention rewards, resulting in reduced hallucinations and more precise event localization on datasets like RexTime and VidHalluc.
Video reasoning requires understanding the causal relationships between events in a video. However, such relationships are often implicit and costly to annotate manually. While existing multimodal large language models (MLLMs) often infer event relations through dense captions or video summaries for video reasoning, such modeling still lacks causal understanding. Without explicit causal structure modeling within and across video events, these models suffer from hallucinations during the video reasoning. In this work, we propose GraphThinker, a reinforcement finetuning-based method that constructs structural event-level scene graphs and enhances visual grounding to jointly reduce hallucinations in video reasoning. Specifically, we first employ an MLLM to construct an event-based video scene graph (EVSG) that explicitly models both intra- and inter-event relations, and incorporate these formed scene graphs into the MLLM as an intermediate thinking process. We also introduce a visual attention reward during reinforcement finetuning, which strengthens video grounding and further mitigates hallucinations. We evaluate GraphThinker on two datasets, RexTime and VidHalluc, where it shows superior ability to capture object and event relations with more precise event localization, reducing hallucinations in video reasoning compared to prior methods.