OE3DIS: Open-Ended 3D Point Cloud Instance Segmentation
This work addresses the problem of restricted autonomy for agents in 3D scene understanding by enabling more flexible segmentation without predefined classes, representing an incremental advance in the field.
The paper tackles the limitation of open-vocab 3D instance segmentation methods that require predefined class names during testing by proposing an open-ended approach that eliminates this need, achieving significant performance improvements over baselines on ScanNet200 and ScanNet++ datasets and surpassing the state-of-the-art Open3DIS method.
Open-Vocab 3D Instance Segmentation methods (OV-3DIS) have recently demonstrated their ability to generalize to unseen objects. However, these methods still depend on predefined class names during testing, restricting the autonomy of agents. To mitigate this constraint, we propose a novel problem termed Open-Ended 3D Instance Segmentation (OE-3DIS), which eliminates the necessity for predefined class names during testing. Moreover, we contribute a comprehensive set of strong baselines, derived from OV-3DIS approaches and leveraging 2D Multimodal Large Language Models. To assess the performance of our OE-3DIS system, we introduce a novel Open-Ended score, evaluating both the semantic and geometric quality of predicted masks and their associated class names, alongside the standard AP score. Our approach demonstrates significant performance improvements over the baselines on the ScanNet200 and ScanNet++ datasets. Remarkably, our method surpasses the performance of Open3DIS, the current state-of-the-art method in OV-3DIS, even in the absence of ground-truth object class names.