Learning from SAM: Harnessing a Foundation Model for Sim2Real Adaptation by Regularization
This addresses the problem of scarce annotated data in robotics for domain adaptation, offering a method that is incremental by building on existing foundation models and unsupervised learning techniques.
The paper tackles domain adaptation for semantic segmentation in robotics by using the Segment Anything Model to obtain segment information on unannotated target data and applying an invariance-variance loss for regularization, achieving performance that outperforms prior work and even matches a network trained with real annotations on the YCB-Video dataset.
Domain adaptation is especially important for robotics applications, where target domain training data is usually scarce and annotations are costly to obtain. We present a method for self-supervised domain adaptation for the scenario where annotated source domain data (e.g. from synthetic generation) is available, but the target domain data is completely unannotated. Our method targets the semantic segmentation task and leverages a segmentation foundation model (Segment Anything Model) to obtain segment information on unannotated data. We take inspiration from recent advances in unsupervised local feature learning and propose an invariance-variance loss over the detected segments for regularizing feature representations in the target domain. Crucially, this loss structure and network architecture can handle overlapping segments and oversegmentation as produced by Segment Anything. We demonstrate the advantage of our method on the challenging YCB-Video and HomebrewedDB datasets and show that it outperforms prior work and, on YCB-Video, even a network trained with real annotations. Additionally, we provide insight through model ablations and show applicability to a custom robotic application.