OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation
This addresses the problem of limited vocabulary in 3D scene understanding for researchers and practitioners, enabling more flexible and powerful 3D AI applications, though it is incremental in combining existing 2D models with 3D data.
The paper tackles 3D open-vocabulary scene understanding by introducing OpenIns3D, a framework that uses a 'Mask-Snap-Lookup' scheme to leverage 2D vision-language models for 3D tasks, achieving state-of-the-art performance across recognition, object detection, and instance segmentation on indoor and outdoor datasets.
In this work, we introduce OpenIns3D, a new 3D-input-only framework for 3D open-vocabulary scene understanding. The OpenIns3D framework employs a "Mask-Snap-Lookup" scheme. The "Mask" module learns class-agnostic mask proposals in 3D point clouds, the "Snap" module generates synthetic scene-level images at multiple scales and leverages 2D vision-language models to extract interesting objects, and the "Lookup" module searches through the outcomes of "Snap" to assign category names to the proposed masks. This approach, yet simple, achieves state-of-the-art performance across a wide range of 3D open-vocabulary tasks, including recognition, object detection, and instance segmentation, on both indoor and outdoor datasets. Moreover, OpenIns3D facilitates effortless switching between different 2D detectors without requiring retraining. When integrated with powerful 2D open-world models, it achieves excellent results in scene understanding tasks. Furthermore, when combined with LLM-powered 2D models, OpenIns3D exhibits an impressive capability to comprehend and process highly complex text queries that demand intricate reasoning and real-world knowledge. Project page: https://zheninghuang.github.io/OpenIns3D/