LightEMMA: Lightweight End-to-End Multimodal Model for Autonomous Driving
This work addresses the need for better evaluation and integration tools for autonomous driving researchers, though it is incremental as it builds on existing VLM methods without introducing a new paradigm.
The authors tackled the lack of a practical platform for evaluating Vision-Language Models (VLMs) in end-to-end autonomous driving by introducing LightEMMA, a lightweight framework that enables dynamic updates and fair comparisons, and found that VLMs show strong interpretation but practical performance remains a concern, with increased complexity not necessarily improving results.
Vision-Language Models (VLMs) have demonstrated significant potential for end-to-end autonomous driving. However, the field still lacks a practical platform that enables dynamic model updates, rapid validation, fair comparison, and intuitive performance assessment. To that end, we introduce LightEMMA, a Lightweight End-to-End Multimodal Model for Autonomous driving. LightEMMA provides a unified, VLM-based autonomous driving framework without ad hoc customizations, enabling easy integration with evolving state-of-the-art commercial and open-source models. We construct twelve autonomous driving agents using various VLMs and evaluate their performance on the challenging nuScenes prediction task, comprehensively assessing computational metrics and providing critical insights. Illustrative examples show that, although VLMs exhibit strong scenario interpretation capabilities, their practical performance in autonomous driving tasks remains a concern. Additionally, increased model complexity and extended reasoning do not necessarily lead to better performance, emphasizing the need for further improvements and task-specific designs. The code is available at https://github.com/michigan-traffic-lab/LightEMMA.