Bridging Vision Language Models and Symbolic Grounding for Video Question Answering
This addresses the challenge of enhancing reasoning capabilities in video understanding for AI systems, but it is incremental as it builds on existing vision language models with symbolic grounding.
The paper tackled the problem of weak temporal grounding and limited interpretability in video question answering by integrating symbolic scene graphs with vision language models, resulting in improved causal and temporal reasoning across benchmarks like NExT-QA, iVQA, and ActivityNet-QA, though gains over strong VLMs were limited.
Video Question Answering (VQA) requires models to reason over spatial, temporal, and causal cues in videos. Recent vision language models (VLMs) achieve strong results but often rely on shallow correlations, leading to weak temporal grounding and limited interpretability. We study symbolic scene graphs (SGs) as intermediate grounding signals for VQA. SGs provide structured object-relation representations that complement VLMs holistic reasoning. We introduce SG-VLM, a modular framework that integrates frozen VLMs with scene graph grounding via prompting and visual localization. Across three benchmarks (NExT-QA, iVQA, ActivityNet-QA) and multiple VLMs (QwenVL, InternVL), SG-VLM improves causal and temporal reasoning and outperforms prior baselines, though gains over strong VLMs are limited. These findings highlight both the promise and current limitations of symbolic grounding, and offer guidance for future hybrid VLM-symbolic approaches in video understanding.