Zero-Shot Scene Understanding with Multimodal Large Language Models for Automated Vehicles
This work addresses scene understanding for autonomous vehicles to enhance explainability and communication, but it is incremental as it primarily benchmarks existing models without introducing new methods.
The paper evaluated four multimodal large language models (MLLMs) for zero-shot scene understanding in autonomous driving, finding that GPT-4o outperformed smaller models but with a modest gap, and ensemble methods showed mixed results with some attributes improving in F1-score while others declined.
Scene understanding is critical for various downstream tasks in autonomous driving, including facilitating driver-agent communication and enhancing human-centered explainability of autonomous vehicle (AV) decisions. This paper evaluates the capability of four multimodal large language models (MLLMs), including relatively small models, to understand scenes in a zero-shot, in-context learning setting. Additionally, we explore whether combining these models using an ensemble approach with majority voting can enhance scene understanding performance. Our experiments demonstrate that GPT-4o, the largest model, outperforms the others in scene understanding. However, the performance gap between GPT-4o and the smaller models is relatively modest, suggesting that advanced techniques such as improved in-context learning, retrieval-augmented generation (RAG), or fine-tuning could further optimize the smaller models' performance. We also observe mixed results with the ensemble approach: while some scene attributes show improvement in performance metrics such as F1-score, others experience a decline. These findings highlight the need for more sophisticated ensemble techniques to achieve consistent gains across all scene attributes. This study underscores the potential of leveraging MLLMs for scene understanding and provides insights into optimizing their performance for autonomous driving applications.