Assessing Situational and Spatial Awareness of VLMs with Synthetically Generated Video
This work addresses the challenge of improving spatial reasoning in VLMs for applications requiring subtle temporal or geometric understanding, though it is incremental as it focuses on diagnostic benchmarking rather than novel solutions.
The researchers tackled the problem of fragile spatial reasoning in vision language models (VLMs) by introducing a synthetic benchmark to assess situational and spatial awareness, finding that performance was only slightly above chance across tasks, with a simple aid like stable color cues providing partial improvement but not resolving underlying weaknesses.
Spatial reasoning in vision language models (VLMs) remains fragile when semantics hinge on subtle temporal or geometric cues. We introduce a synthetic benchmark that probes two complementary skills: situational awareness (recognizing whether an interaction is harmful or benign) and spatial awareness (tracking who does what to whom, and reasoning about relative positions and motion). Through minimal video pairs, we test three challenges: distinguishing violence from benign activity, binding assailant roles across viewpoints, and judging fine-grained trajectory alignment. While we evaluate recent VLMs in a training-free setting, the benchmark is applicable to any video classification model. Results show performance only slightly above chance across tasks. A simple aid, stable color cues, partly reduces assailant role confusions but does not resolve the underlying weakness. By releasing data and code, we aim to provide reproducible diagnostics and seed exploration of lightweight spatial priors to complement large-scale pretraining.