GroundFlow: A Plug-in Module for Temporal Reasoning on 3D Point Cloud Sequential Grounding
This addresses the challenge of temporal reasoning in 3D visual grounding for applications like robotics and AR/VR, though it is incremental as it builds on existing 3DVG methods.
The paper tackles the problem of sequential grounding in 3D point clouds (SG3D), where existing methods struggle with temporal reasoning due to pronouns in text instructions, by proposing GroundFlow, a plug-in module that improves baseline accuracy by +7.5% and +10.2% and achieves state-of-the-art performance across five datasets.
Sequential grounding in 3D point clouds (SG3D) refers to locating sequences of objects by following text instructions for a daily activity with detailed steps. Current 3D visual grounding (3DVG) methods treat text instructions with multiple steps as a whole, without extracting useful temporal information from each step. However, the instructions in SG3D often contain pronouns such as "it", "here" and "the same" to make language expressions concise. This requires grounding methods to understand the context and retrieve relevant information from previous steps to correctly locate object sequences. Due to the lack of an effective module for collecting related historical information, state-of-the-art 3DVG methods face significant challenges in adapting to the SG3D task. To fill this gap, we propose GroundFlow -- a plug-in module for temporal reasoning on 3D point cloud sequential grounding. Firstly, we demonstrate that integrating GroundFlow improves the task accuracy of 3DVG baseline methods by a large margin (+7.5\% and +10.2\%) in the SG3D benchmark, even outperforming a 3D large language model pre-trained on various datasets. Furthermore, we selectively extract both short-term and long-term step information based on its relevance to the current instruction, enabling GroundFlow to take a comprehensive view of historical information and maintain its temporal understanding advantage as step counts increase. Overall, our work introduces temporal reasoning capabilities to existing 3DVG models and achieves state-of-the-art performance in the SG3D benchmark across five datasets.