AccidentBench: Benchmarking Multimodal Understanding and Reasoning in Vehicle Accidents and Beyond
This benchmark addresses the need for rigorous testing of multimodal AI models in safety-critical scenarios like traffic, air, and water navigation, which is crucial for developing safer and more robust AI systems, though it is incremental as it builds on existing benchmarking efforts.
The authors tackled the problem of evaluating multimodal models in safety-critical, dynamic real-world settings by introducing AccidentBench, a large-scale benchmark with over 19,000 question-answer pairs across vehicle accidents and other domains, and found that state-of-the-art models achieve only about 18% accuracy on the hardest tasks.
Rapid advances in multimodal models demand benchmarks that rigorously evaluate understanding and reasoning in safety-critical, dynamic real-world settings. We present AccidentBench, a large-scale benchmark that combines vehicle accident scenarios with Beyond domains, safety-critical settings in air and water that emphasize spatial and temporal reasoning (e.g., navigation, orientation, multi-vehicle motion). The benchmark contains approximately 2000 videos and over 19000 human-annotated question--answer pairs spanning multiple video lengths (short/medium/long) and difficulty levels (easy/medium/hard). Tasks systematically probe core capabilities: temporal, spatial, and intent understanding and reasoning. By unifying accident-centric traffic scenes with broader safety-critical scenarios in air and water, AccidentBench offers a comprehensive, physically grounded testbed for evaluating models under real-world variability. Evaluations of state-of-the-art models (e.g., Gemini-2.5 Pro and GPT-5) show that even the strongest models achieve only about 18% accuracy on the hardest tasks and longest videos, revealing substantial gaps in real-world temporal, spatial, and intent reasoning. AccidentBench is designed to expose these critical gaps and drive the development of multimodal models that are safer, more robust, and better aligned with real-world safety-critical challenges. The code and dataset are available at: https://github.com/SafeRL-Lab/AccidentBench