CutS3D: Cutting Semantics in 3D for 2D Unsupervised Instance Segmentation
This work addresses the problem of improving unsupervised instance segmentation for computer vision applications, representing an incremental advancement over existing methods.
The paper tackles the problem of unsupervised instance segmentation in 2D images by addressing failures in separating overlapping instances, using 3D point clouds to cut semantic masks and introducing a Spatial Importance function and Spatial Confidence components. The result is that their approach outperforms competing methods on multiple standard benchmarks for unsupervised instance segmentation and object detection.
Traditionally, algorithms that learn to segment object instances in 2D images have heavily relied on large amounts of human-annotated data. Only recently, novel approaches have emerged tackling this problem in an unsupervised fashion. Generally, these approaches first generate pseudo-masks and then train a class-agnostic detector. While such methods deliver the current state of the art, they often fail to correctly separate instances overlapping in 2D image space since only semantics are considered. To tackle this issue, we instead propose to cut the semantic masks in 3D to obtain the final 2D instances by utilizing a point cloud representation of the scene. Furthermore, we derive a Spatial Importance function, which we use to resharpen the semantics along the 3D borders of instances. Nevertheless, these pseudo-masks are still subject to mask ambiguity. To address this issue, we further propose to augment the training of a class-agnostic detector with three Spatial Confidence components aiming to isolate a clean learning signal. With these contributions, our approach outperforms competing methods across multiple standard benchmarks for unsupervised instance segmentation and object detection.