3D Guided Weakly Supervised Semantic Segmentation
This work aims to reduce the annotation burden for semantic segmentation, benefiting researchers and practitioners in computer vision who deal with large datasets.
This paper addresses the high cost of pixel-wise annotations for semantic segmentation by proposing a weakly supervised 2D semantic segmentation model. It leverages sparse bounding box labels and available 3D information to generate accurate pixel-wise segment proposals, which are then used as pseudo labels to train a segmentation network. The method demonstrates effectiveness on the 2D-3D-S dataset, generating accurate proposals even with bounding box labels on only a small subset of training images.
Pixel-wise clean annotation is necessary for fully-supervised semantic segmentation, which is laborious and expensive to obtain. In this paper, we propose a weakly supervised 2D semantic segmentation model by incorporating sparse bounding box labels with available 3D information, which is much easier to obtain with advanced sensors. We manually labeled a subset of the 2D-3D Semantics(2D-3D-S) dataset with bounding boxes, and introduce our 2D-3D inference module to generate accurate pixel-wise segment proposal masks. Guided by 3D information, we first generate a point cloud of objects and calculate objectness probability score for each point. Then we project the point cloud with objectness probabilities back to 2D images followed by a refinement step to obtain segment proposals, which are treated as pseudo labels to train a semantic segmentation network. Our method works in a recursive manner to gradually refine the above-mentioned segment proposals. Extensive experimental results on the 2D-3D-S dataset show that the proposed method can generate accurate segment proposals when bounding box labels are available on only a small subset of training images. Performance comparison with recent state-of-the-art methods further illustrates the effectiveness of our method.