Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models
This addresses the problem of 3D scene understanding without labeled data for researchers and applications in robotics and AR/VR, representing a novel method for a known bottleneck.
The paper tackles open-vocabulary 3D semantic segmentation by leveraging pre-trained text-to-image diffusion models, achieving a 12% improvement over state-of-the-art methods on ScanNet200.
In this paper, we investigate the use of diffusion models which are pre-trained on large-scale image-caption pairs for open-vocabulary 3D semantic understanding. We propose a novel method, namely Diff2Scene, which leverages frozen representations from text-image generative models, along with salient-aware and geometric-aware masks, for open-vocabulary 3D semantic segmentation and visual grounding tasks. Diff2Scene gets rid of any labeled 3D data and effectively identifies objects, appearances, materials, locations and their compositions in 3D scenes. We show that it outperforms competitive baselines and achieves significant improvements over state-of-the-art methods. In particular, Diff2Scene improves the state-of-the-art method on ScanNet200 by 12%.