SegPoint: Segment Any Point Cloud via Large Language Model
This work addresses the need for a versatile 3D segmentation model that can infer implicit instructions, benefiting researchers and practitioners in computer vision and robotics, though it is incremental as it builds on existing LLM and segmentation methods.
The paper tackles the problem of 3D point cloud segmentation lacking a unified framework for implicit user intentions by proposing SegPoint, which leverages a multi-modal LLM to handle multiple segmentation tasks, achieving competitive results on benchmarks like ScanRefer and ScanNet and outstanding performance on the new Instruct3D dataset with 2,565 pairs.
Despite significant progress in 3D point cloud segmentation, existing methods primarily address specific tasks and depend on explicit instructions to identify targets, lacking the capability to infer and understand implicit user intentions in a unified framework. In this work, we propose a model, called SegPoint, that leverages the reasoning capabilities of a multi-modal Large Language Model (LLM) to produce point-wise segmentation masks across a diverse range of tasks: 1) 3D instruction segmentation, 2) 3D referring segmentation, 3) 3D semantic segmentation, and 4) 3D open-vocabulary semantic segmentation. To advance 3D instruction research, we introduce a new benchmark, Instruct3D, designed to evaluate segmentation performance from complex and implicit instructional texts, featuring 2,565 point cloud-instruction pairs. Our experimental results demonstrate that SegPoint achieves competitive performance on established benchmarks such as ScanRefer for referring segmentation and ScanNet for semantic segmentation, while delivering outstanding outcomes on the Instruct3D dataset. To our knowledge, SegPoint is the first model to address these varied segmentation tasks within a single framework, achieving satisfactory performance.