Know-Show: Benchmarking Video-Language Models on Spatio-Temporal Grounded Reasoning
This work addresses the challenge of developing interpretable and reliable multimodal reasoning systems for video-language understanding, though it is incremental as it builds on existing benchmarks and models.
The authors tackled the problem of weakly grounded spatio-temporal reasoning in Video-Language Models by introducing Know-Show, a benchmark with 2.5K human-authored questions across five scenarios, which revealed significant gaps between current models and human reasoning, and they proposed GRAM, a training-free plug-in that improved performance, though models still struggled with fine-grained interactions.
Large Video-Language Models (Video-LMs) have achieved impressive progress in multimodal understanding, yet their reasoning remains weakly grounded in space and time. We present Know-Show, a new benchmark designed to evaluate spatio-temporal grounded reasoning, the ability of a model to reason about actions and their semantics while simultaneously grounding its inferences in visual and temporal evidence. Know-Show unifies reasoning and localization within a single evaluation framework consisting of five complementary scenarios across spatial (person, object, person-object, and hand-object) and temporal dimensions. Built from Charades, Action Genome, and Ego4D with 2.5K human-authored questions, the benchmark exposes significant gaps between current Video-LMs and human reasoning. To bridge this gap, we propose GRAM, a training-free plug-in that augments Video-LMs with fine-grained grounding through attention-based video token selection and explicit timestamp encoding. Extensive experiments across open and closed Video-LMs (Qwen, VideoLLaVA, GPT-4o, and Gemini, etc.) reveal that existing models struggle to "show what they know" and vice versa, especially in fine-grained hand-object interactions. Know-Show establishes a unified standard for assessing grounded reasoning in video-language understanding and provides insights toward developing interpretable and reliable multimodal reasoning systems. We will release the dataset and the code at https://github.com/LUNAProject22/Know-Show.