RGB-based Semantic Segmentation Using Self-Supervised Depth Pre-Training
This work addresses the scalability issue in pre-training for semantic segmentation, offering a more efficient alternative to manual annotation, though it is incremental as it builds on existing self-supervised and pre-training techniques.
The paper tackles the problem of reducing reliance on large manually annotated datasets for pre-training semantic segmentation models by proposing a self-supervised method using automatically generated depth-based labels (HN-labels). The result shows that this approach uses 25x fewer images than ImageNet pre-training and achieves better segmentation accuracy on NYUv2 and CamVid datasets.
Although well-known large-scale datasets, such as ImageNet, have driven image understanding forward, most of these datasets require extensive manual annotation and are thus not easily scalable. This limits the advancement of image understanding techniques. The impact of these large-scale datasets can be observed in almost every vision task and technique in the form of pre-training for initialization. In this work, we propose an easily scalable and self-supervised technique that can be used to pre-train any semantic RGB segmentation method. In particular, our pre-training approach makes use of automatically generated labels that can be obtained using depth sensors. These labels, denoted by HN-labels, represent different height and normal patches, which allow mining of local semantic information that is useful in the task of semantic RGB segmentation. We show how our proposed self-supervised pre-training with HN-labels can be used to replace ImageNet pre-training, while using 25x less images and without requiring any manual labeling. We pre-train a semantic segmentation network with our HN-labels, which resembles our final task more than pre-training on a less related task, e.g. classification with ImageNet. We evaluate on two datasets (NYUv2 and CamVid), and we show how the similarity in tasks is advantageous not only in speeding up the pre-training process, but also in achieving better final semantic segmentation accuracy than ImageNet pre-training