Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding
This addresses the limitation of needing large labeled datasets for 3D object localization from text, offering a more flexible solution for applications in robotics or augmented reality.
The paper tackles the problem of 3D visual grounding, which requires extensive annotations and predefined vocabularies, by proposing a visual programming approach using large language models for zero-shot open-vocabulary scenarios, achieving performance that outperforms some supervised baselines.
3D Visual Grounding (3DVG) aims at localizing 3D object based on textual descriptions. Conventional supervised methods for 3DVG often necessitate extensive annotations and a predefined vocabulary, which can be restrictive. To address this issue, we propose a novel visual programming approach for zero-shot open-vocabulary 3DVG, leveraging the capabilities of large language models (LLMs). Our approach begins with a unique dialog-based method, engaging with LLMs to establish a foundational understanding of zero-shot 3DVG. Building on this, we design a visual program that consists of three types of modules, i.e., view-independent, view-dependent, and functional modules. These modules, specifically tailored for 3D scenarios, work collaboratively to perform complex reasoning and inference. Furthermore, we develop an innovative language-object correlation module to extend the scope of existing 3D object detectors into open-vocabulary scenarios. Extensive experiments demonstrate that our zero-shot approach can outperform some supervised baselines, marking a significant stride towards effective 3DVG.