Embodied Scene Understanding for Vision Language Models via MetaVQA
This addresses the need for better evaluation of VLMs as embodied AI agents in mobility applications, though it is incremental as it builds on existing VQA and simulation methods.
The authors tackled the lack of standardized benchmarks for evaluating spatial reasoning in Vision Language Models (VLMs) by creating MetaVQA, a comprehensive benchmark using Visual Question Answering and closed-loop simulations. Their experiments showed that fine-tuning VLMs with MetaVQA significantly improved spatial reasoning and embodied scene comprehension, with enhanced VQA accuracies and safety-aware driving maneuvers, and demonstrated strong transferability from simulation to real-world observation.
Vision Language Models (VLMs) demonstrate significant potential as embodied AI agents for various mobility applications. However, a standardized, closed-loop benchmark for evaluating their spatial reasoning and sequential decision-making capabilities is lacking. To address this, we present MetaVQA: a comprehensive benchmark designed to assess and enhance VLMs' understanding of spatial relationships and scene dynamics through Visual Question Answering (VQA) and closed-loop simulations. MetaVQA leverages Set-of-Mark prompting and top-down view ground-truth annotations from nuScenes and Waymo datasets to automatically generate extensive question-answer pairs based on diverse real-world traffic scenarios, ensuring object-centric and context-rich instructions. Our experiments show that fine-tuning VLMs with the MetaVQA dataset significantly improves their spatial reasoning and embodied scene comprehension in safety-critical simulations, evident not only in improved VQA accuracies but also in emerging safety-aware driving maneuvers. In addition, the learning demonstrates strong transferability from simulation to real-world observation. Code and data will be publicly available at https://metadriverse.github.io/metavqa .