CrashChat: A Multimodal Large Language Model for Multitask Traffic Crash Video Analysis
This work addresses the need for a unified tool to analyze crash videos for traffic safety research and autonomous driving accountability, offering an incremental improvement by adapting existing MLLM frameworks to a specific domain.
The paper tackles the problem of automating multitask traffic crash video analysis, which includes crash recognition, temporal grounding, and high-level understanding, by proposing CrashChat, a multimodal large language model. It achieves state-of-the-art performance with near-perfect accuracy in crash recognition, a 176% improvement in crash localization, and a 40% improvement in pre-crash localization, along with significant gains in textual metrics like BLEU and ROUGE scores.
Automating crash video analysis is essential to leverage the growing availability of driving video data for traffic safety research and accountability attribution in autonomous driving. Crash video analysis is a challenging multitask problem due to the complex spatiotemporal dynamics of crash events in video data and the diverse analytical requirements involved. It requires capabilities spanning crash recognition, temporal grounding, and high-level video understanding. Existing models, however, cannot perform all these tasks within a unified framework, and effective training strategies for such models remain underexplored. To fill these gaps, this paper proposes CrashChat, a multimodal large language model (MLLM) for multitask traffic crash analysis, built upon VideoLLaMA3. CrashChat acquires domain-specific knowledge through instruction fine-tuning and employs a novel multitask learning strategy based on task decoupling and grouping, which maximizes the benefit of joint learning within and across task groups while mitigating negative transfer. Numerical experiments on consolidated public datasets demonstrate that CrashChat consistently outperforms existing MLLMs across model scales and traditional vision-based methods, achieving state-of-the-art performance. It reaches near-perfect accuracy in crash recognition, a 176\% improvement in crash localization, and a 40\% improvement in the more challenging pre-crash localization. Compared to general MLLMs, it substantially enhances textual accuracy and content coverage in crash description and reasoning tasks, with 0.18-0.41 increases in BLEU scores and 0.18-0.42 increases in ROUGE scores. Beyond its strong performance, CrashChat is a convenient, end-to-end analytical tool ready for practical implementation. The dataset and implementation code for CrashChat are available at https://github.com/Liangkd/CrashChat.