VRU-Accident: A Vision-Language Benchmark for Video Question Answering and Dense Captioning for Accident Scene Understanding
This work addresses the problem of benchmarking MLLMs for VRU safety in autonomous driving, providing a domain-specific tool for researchers and developers, but it is incremental as it builds on existing multimodal evaluation frameworks.
The paper tackles the lack of a standardized benchmark for evaluating multimodal large language models (MLLMs) in safety-critical autonomous driving scenarios involving vulnerable road users (VRUs), by introducing VRU-Accident, a large-scale vision-language benchmark with 1K accident videos, 6K question-answer pairs, and 1K dense descriptions, and finds that MLLMs struggle with reasoning about accident causes, types, and preventability despite performing well on visual attributes.
Ensuring the safety of vulnerable road users (VRUs), such as pedestrians and cyclists, is a critical challenge for autonomous driving systems, as crashes involving VRUs often result in severe or fatal consequences. While multimodal large language models (MLLMs) have shown promise in enhancing scene understanding and decision making in autonomous vehicles, there is currently no standardized benchmark to quantitatively evaluate their reasoning abilities in complex, safety-critical scenarios involving VRUs. To address this gap, we present VRU-Accident, a large-scale vision-language benchmark designed to evaluate MLLMs in high-risk traffic scenarios involving VRUs. VRU-Accident comprises 1K real-world dashcam accident videos, annotated with 6K multiple-choice question-answer pairs across six safety-critical categories (with 24K candidate options and 3.4K unique answer choices), as well as 1K dense scene descriptions. Unlike prior works, our benchmark focuses explicitly on VRU-vehicle accidents, providing rich, fine-grained annotations that capture both spatial-temporal dynamics and causal semantics of accidents. To assess the current landscape of MLLMs, we conduct a comprehensive evaluation of 17 state-of-the-art models on the multiple-choice VQA task and on the dense captioning task. Our findings reveal that while MLLMs perform reasonably well on visually grounded attributes, they face significant challenges in reasoning and describing accident causes, types, and preventability.