Segment Any 3D-Part in a Scene from a Sentence
This work addresses the challenge of fine-grained 3D scene understanding for applications like robotics and AR/VR, though it is incremental as it builds on existing object-level methods.
The paper tackles the problem of segmenting any 3D part in a scene from natural language descriptions, overcoming data and methodological limitations by introducing the 3D-PU dataset and the OpenPart3D framework, achieving strong generalization and superiority in open-vocabulary 3D part-level segmentation tasks.
This paper aims to achieve the segmentation of any 3D part in a scene based on natural language descriptions, extending beyond traditional object-level 3D scene understanding and addressing both data and methodological challenges. Due to the expensive acquisition and annotation burden, existing datasets and methods are predominantly limited to object-level comprehension. To overcome the limitations of data and annotation availability, we introduce the 3D-PU dataset, the first large-scale 3D dataset with dense part annotations, created through an innovative and cost-effective method for constructing synthetic 3D scenes with fine-grained part-level annotations, paving the way for advanced 3D-part scene understanding. On the methodological side, we propose OpenPart3D, a 3D-input-only framework to effectively tackle the challenges of part-level segmentation. Extensive experiments demonstrate the superiority of our approach in open-vocabulary 3D scene understanding tasks at the part level, with strong generalization capabilities across various 3D scene datasets.