SOVABench: A Vehicle Surveillance Action Retrieval Benchmark for Multimodal Large Language Models
This addresses the problem of evaluating action discrimination in surveillance for researchers and practitioners, though it is incremental as it builds on existing MLLM capabilities.
The authors tackled the lack of action discrimination in video surveillance retrieval by introducing SOVABench, a real-world benchmark for vehicle-related actions, and developed a training-free MLLM-based framework that achieves strong performance on this benchmark and other spatial and counting tasks.
Automatic identification of events and recurrent behavior analysis are critical for video surveillance. However, most existing content-based video retrieval benchmarks focus on scene-level similarity and do not evaluate the action discrimination required in surveillance. To address this gap, we introduce SOVABench (Surveillance Opposite Vehicle Actions Benchmark), a real-world retrieval benchmark built from surveillance footage and centered on vehicle-related actions. SOVABench defines two evaluation protocols (inter-pair and intra-pair) to assess cross-action discrimination and temporal direction understanding. Although action distinctions are generally intuitive for human observers, our experiments show that they remain challenging for state-of-the-art vision and multimodal models. Leveraging the visual reasoning and instruction-following capabilities of Multimodal Large Language Models (MLLMs), we present a training-free framework for producing interpretable embeddings from MLLM-generated descriptions for both images and videos. The framework achieves strong performance on SOVABench as well as on several spatial and counting benchmarks where contrastive Vision-Language Models often fail. The code, annotations, and instructions to construct the benchmark are publicly available.