Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning
This work addresses the need for interpretable autonomous vehicles by providing a dataset for researchers in visual scene understanding, though it is incremental as it builds on existing multimodal datasets.
The paper introduces Rank2Tell, a multimodal dataset for ranking object importance and generating reasons in driving scenes, and presents a joint model that achieves quantitative performance on this benchmark.
The widespread adoption of commercial autonomous vehicles (AVs) and advanced driver assistance systems (ADAS) may largely depend on their acceptance by society, for which their perceived trustworthiness and interpretability to riders are crucial. In general, this task is challenging because modern autonomous systems software relies heavily on black-box artificial intelligence models. Towards this goal, this paper introduces a novel dataset, Rank2Tell, a multi-modal ego-centric dataset for Ranking the importance level and Telling the reason for the importance. Using various close and open-ended visual question answering, the dataset provides dense annotations of various semantic, spatial, temporal, and relational attributes of various important objects in complex traffic scenarios. The dense annotations and unique attributes of the dataset make it a valuable resource for researchers working on visual scene understanding and related fields. Furthermore, we introduce a joint model for joint importance level ranking and natural language captions generation to benchmark our dataset and demonstrate performance with quantitative evaluations.