Self-supervised Training of Proposal-based Segmentation via Background Prediction
This addresses the challenge of expensive data annotation in scenarios where images differ from standard benchmarks, offering a self-supervised solution for object detection and segmentation.
The paper tackles the problem of poor generalization in supervised object detection to visually different images by introducing a self-supervised approach for object detection and segmentation using monocular images from a moving camera, resulting in accurate detections and segmentations that outperform existing self-supervised methods and approach weakly supervised ones.
While supervised object detection methods achieve impressive accuracy, they generalize poorly to images whose appearance significantly differs from the data they have been trained on. To address this in scenarios where annotating data is prohibitively expensive, we introduce a self-supervised approach to object detection and segmentation, able to work with monocular images captured with a moving camera. At the heart of our approach lies the observation that segmentation and background reconstruction are linked tasks, and the idea that, because we observe a structured scene, background regions can be re-synthesized from their surroundings, whereas regions depicting the object cannot. We therefore encode this intuition as a self-supervised loss function that we exploit to train a proposal-based segmentation network. To account for the discrete nature of object proposals, we develop a Monte Carlo-based training strategy that allows us to explore the large space of object proposals. Our experiments demonstrate that our approach yields accurate detections and segmentations in images that visually depart from those of standard benchmarks, outperforming existing self-supervised methods and approaching weakly supervised ones that exploit large annotated datasets.