Few-Click-Driven Interactive 3D Segmentation with Semantic Embedding
This work addresses the need for efficient and generalizable interactive 3D instance segmentation for real-time applications like robotics and rapid annotation, offering a significant improvement over existing sequential or 2D-dependent methods.
The paper proposes a novel interactive 3D segmentation framework that processes multiple object clicks in a single forward pass on sparse 3D points, using a point Transformer encoder and hierarchical mask decoder with semantic embeddings. It achieves over 20% mIoU improvement over baselines and 8-10% gains in cross-dataset evaluation, often requiring only one click per object.
Interactive segmentation allows efficient label generation by leveraging user-provided clicks to progressively refine predictions, which is critical when fully supervised labels are costly or generalization to unseen classes is needed. Existing 3D interactive methods are limited: most operate sequentially, predicting only one object per iteration with binary masks, while several recent approaches depend on 2D foundation models and camera alignment to bridge the 2D-3D gap. To address these limitations, we propose a novel interactive segmentation framework that operates directly on sparse, randomly downsampled 3D points and processes multiple object clicks in a single forward pass. Our framework consists of a point Transformer-based encoder and a hierarchical mask decoder, which integrates multi-level crop-and-merge operations conditioned on learnable semantic embeddings. Unlike prior interactive approaches that require repeated model updates after each manually corrective click, our method jointly reasons over all click queries, modeling inter-instance relationships and refining both spatial masks and semantic predictions through spatial and semantic embeddings. Extensive experiments demonstrate that our model improves the mIoU metric by over 20 percent compared to strong baselines and achieves 8-10 percent gains under cross-dataset evaluation for a one-click per instance setting, often requiring only a single click per object. Our approach provides a generalizable and efficient solution for interactive 3D instance segmentation, particularly suitable for real-time applications such as robotic manipulation, navigation, and rapid 3D semantic annotation.