SceneCOT: Eliciting Grounded Chain-of-Thought Reasoning in 3D Scenes
This addresses the challenge of human-like scene-object grounded reasoning in 3D AI systems, representing an incremental advance by applying Chain-of-Thought reasoning to 3D scenes for the first time.
The paper tackles the problem of achieving grounded question-answering in 3D Large Language Models by introducing SceneCOT, a framework that decouples complex reasoning tasks into simpler problems using visual clues from multimodal expert modules, and it demonstrates strong performance across various 3D scene reasoning benchmarks with high grounding-QA coherence.
Existing research on 3D Large Language Models (LLMs) still struggles to achieve grounded question-answering, primarily due to the under-exploration of the mechanism of human-like scene-object grounded reasoning. This paper bridges the gap by presenting a novel framework. We first introduce a grounded Chain-of-Thought reasoning method in 3D scenes (SCENECOT), decoupling a complex reasoning task into simpler and manageable problems, and building corresponding visual clues based on multimodal expert modules. To enable such a method, we develop SCENECOT-185K, the first large-scale grounded CoT reasoning dataset, consisting of 185K high-quality instances. Extensive experiments across various complex 3D scene reasoning benchmarks demonstrate that our new framework achieves strong performance with high grounding-QA coherence. To the best of our knowledge, this is the first successful application of CoT reasoning to 3D scene understanding, enabling step-by-step human-like reasoning and showing potential for extension to broader 3D scene understanding scenarios.