SAD: Segment Any RGBD
This work addresses the problem of geometry-aware segmentation for computer vision applications, but it is incremental as it builds directly on SAM.
The paper tackles the limitation of the Segment Anything Model (SAM) in over-segmenting RGB images due to its emphasis on texture over geometry, by proposing the Segment Any RGBD (SAD) model that uses SAM to segment rendered depth maps for enhanced geometry cues, achieving 3D panoptic segmentation with open-vocabulary semantic segmentation.
The Segment Anything Model (SAM) has demonstrated its effectiveness in segmenting any part of 2D RGB images. However, SAM exhibits a stronger emphasis on texture information while paying less attention to geometry information when segmenting RGB images. To address this limitation, we propose the Segment Any RGBD (SAD) model, which is specifically designed to extract geometry information directly from images. Inspired by the natural ability of humans to identify objects through the visualization of depth maps, SAD utilizes SAM to segment the rendered depth map, thus providing cues with enhanced geometry information and mitigating the issue of over-segmentation. We further include the open-vocabulary semantic segmentation in our framework, so that the 3D panoptic segmentation is fulfilled. The project is available on https://github.com/Jun-CEN/SegmentAnyRGBD.