Driver Assistance System Based on Multimodal Data Hazard Detection
This work addresses the problem of enhancing driving safety for autonomous vehicles by improving hazard detection, though it is incremental as it builds on existing multimodal approaches.
The paper tackles the challenge of detecting driving anomalies by proposing a multimodal driver assistance system that integrates road condition video, driver facial video, and audio data, resulting in improved incident recognition accuracy and reduced misjudgments.
Autonomous driving technology has advanced significantly, yet detecting driving anomalies remains a major challenge due to the long-tailed distribution of driving events. Existing methods primarily rely on single-modal road condition video data, which limits their ability to capture rare and unpredictable driving incidents. This paper proposes a multimodal driver assistance detection system that integrates road condition video, driver facial video, and audio data to enhance incident recognition accuracy. Our model employs an attention-based intermediate fusion strategy, enabling end-to-end learning without separate feature extraction. To support this approach, we develop a new three-modality dataset using a driving simulator. Experimental results demonstrate that our method effectively captures cross-modal correlations, reducing misjudgments and improving driving safety.