Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers
This work addresses deficiencies in general referencing and grounding for intricate 3D scene comprehension, benefiting applications in robotics and augmented reality, though it is incremental as it builds on existing 3D LLM methods.
The paper tackles the problem of 3D scene understanding by introducing object identifiers and object-centric representations to improve referencing and grounding capabilities, resulting in significant performance gains on multiple benchmarks like ScanRefer and ScanQA with minimal fine-tuning.
Recent advancements in 3D Large Language Models (LLMs) have demonstrated promising capabilities for 3D scene understanding. However, previous methods exhibit deficiencies in general referencing and grounding capabilities for intricate scene comprehension. In this paper, we introduce the use of object identifiers and object-centric representations to interact with scenes at the object level. Specifically, we decompose the input 3D scene into a set of object proposals, each assigned a unique identifier token, which enables efficient object referencing and grounding during user-assistant interactions. Given the scarcity of scene-language data, we model the scene embeddings as a sequence of explicit object-level embeddings, derived from semantic-rich 2D or 3D representations. By employing object identifiers, we transform diverse 3D scene-language tasks into a unified question-answering format, facilitating joint training without the need for additional task-specific heads. With minimal fine-tuning on all downstream tasks, our model significantly outperforms existing methods on benchmarks including ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, and SQA3D.