Video Event Reasoning and Prediction by Fusing World Knowledge from LLMs with Vision Foundation Models
This work addresses the problem of enabling AI systems to perform advanced reasoning and prediction in video understanding, with potential applications in robotics and human-computer interaction, though it is incremental as it builds on existing models like VFMs and LLMs.
The paper tackles the limitation of video understanding models in high-level cognitive tasks like causal reasoning and future prediction by proposing a framework that fuses a Vision Foundation Model with a Large Language Model, achieving state-of-the-art performance on multiple benchmarks and demonstrating zero-shot generalization to unseen reasoning tasks.
Current video understanding models excel at recognizing "what" is happening but fall short in high-level cognitive tasks like causal reasoning and future prediction, a limitation rooted in their lack of commonsense world knowledge. To bridge this cognitive gap, we propose a novel framework that synergistically fuses a powerful Vision Foundation Model (VFM) for deep visual perception with a Large Language Model (LLM) serving as a knowledge-driven reasoning core. Our key technical innovation is a sophisticated fusion module, inspired by the Q-Former architecture, which distills complex spatiotemporal and object-centric visual features into a concise, language-aligned representation. This enables the LLM to effectively ground its inferential processes in direct visual evidence. The model is trained via a two-stage strategy, beginning with large-scale alignment pre-training on video-text data, followed by targeted instruction fine-tuning on a curated dataset designed to elicit advanced reasoning and prediction skills. Extensive experiments demonstrate that our model achieves state-of-the-art performance on multiple challenging benchmarks. Notably, it exhibits remarkable zero-shot generalization to unseen reasoning tasks, and our in-depth ablation studies validate the critical contribution of each architectural component. This work pushes the boundary of machine perception from simple recognition towards genuine cognitive understanding, paving the way for more intelligent and capable AI systems in robotics, human-computer interaction, and beyond.