MPDrive: Improving Spatial Understanding with Marker-Based Prompt Learning for Autonomous Driving
This work addresses spatial understanding challenges in autonomous driving VQA, offering a novel method to improve accuracy, but it is incremental as it builds on existing frameworks with a specific enhancement.
The paper tackles the problem of semantic gaps in spatial understanding for autonomous driving visual question answering by proposing MPDrive, a marker-based prompt learning framework that uses visual markers to represent spatial coordinates, achieving state-of-the-art performance on datasets like DriveLM and CODA-LM.
Autonomous driving visual question answering (AD-VQA) aims to answer questions related to perception, prediction, and planning based on given driving scene images, heavily relying on the model's spatial understanding capabilities. Prior works typically express spatial information through textual representations of coordinates, resulting in semantic gaps between visual coordinate representations and textual descriptions. This oversight hinders the accurate transmission of spatial information and increases the expressive burden. To address this, we propose a novel Marker-based Prompt learning framework (MPDrive), which represents spatial coordinates by concise visual markers, ensuring linguistic expressive consistency and enhancing the accuracy of both visual perception and spatial expression in AD-VQA. Specifically, we create marker images by employing a detection expert to overlay object regions with numerical labels, converting complex textual coordinate generation into straightforward text-based visual marker predictions. Moreover, we fuse original and marker images as scene-level features and integrate them with detection priors to derive instance-level features. By combining these features, we construct dual-granularity visual prompts that stimulate the LLM's spatial perception capabilities. Extensive experiments on the DriveLM and CODA-LM datasets show that MPDrive achieves state-of-the-art performance, particularly in cases requiring sophisticated spatial understanding.