Empowering Large Language Models with 3D Situation Awareness
This addresses a key limitation in applying LLMs to 3D scenes for tasks like robotics or AR/VR, though it is incremental as it builds on existing LLM and VLM techniques.
The paper tackles the problem of large language models lacking egocentric perspective in 3D scene understanding by proposing a method to automatically generate situation-aware datasets and a grounding module, resulting in enhanced 3D situational awareness and expanded datasets with reduced manual effort.
Driven by the great success of Large Language Models (LLMs) in the 2D image domain, their applications in 3D scene understanding has emerged as a new trend. A key difference between 3D and 2D is that the situation of an egocentric observer in 3D scenes can change, resulting in different descriptions (e.g., ''left" or ''right"). However, current LLM-based methods overlook the egocentric perspective and simply use datasets from a global viewpoint. To address this issue, we propose a novel approach to automatically generate a situation-aware dataset by leveraging the scanning trajectory during data collection and utilizing Vision-Language Models (VLMs) to produce high-quality captions and question-answer pairs. Furthermore, we introduce a situation grounding module to explicitly predict the position and orientation of observer's viewpoint, thereby enabling LLMs to ground situation description in 3D scenes. We evaluate our approach on several benchmarks, demonstrating that our method effectively enhances the 3D situational awareness of LLMs while significantly expanding existing datasets and reducing manual effort.