A Benchmark for Crime Surveillance Video Analysis with Large Models
This provides a new benchmark for researchers in computer vision and surveillance to assess MLLMs on anomaly detection, but it is incremental as it adapts existing datasets and methods.
The authors tackled the lack of benchmarks for evaluating multimodal large language models (MLLMs) in crime surveillance video analysis by proposing UCVL, a new benchmark with 1,829 videos and diverse QA pairs, and benchmarked eight MLLMs, showing reliability and improved performance after fine-tuning.
Anomaly analysis in surveillance videos is a crucial topic in computer vision. In recent years, multimodal large language models (MLLMs) have outperformed task-specific models in various domains. Although MLLMs are particularly versatile, their abilities to understand anomalous concepts and details are insufficiently studied because of the outdated benchmarks of this field not providing MLLM-style QAs and efficient algorithms to assess the model's open-ended text responses. To fill this gap, we propose a benchmark for crime surveillance video analysis with large models denoted as UCVL, including 1,829 videos and reorganized annotations from the UCF-Crime and UCF-Crime Annotation datasets. We design six types of questions and generate diverse QA pairs. Then we develop detailed instructions and use OpenAI's GPT-4o for accurate assessment. We benchmark eight prevailing MLLMs ranging from 0.5B to 40B parameters, and the results demonstrate the reliability of this bench. Moreover, we finetune LLaVA-OneVision on UCVL's training set. The improvement validates our data's high quality for video anomaly analysis.