Unseen Object Instance Segmentation with Fully Test-time RGB-D Embeddings Adaptation
This work addresses the domain shift challenge for robots in new environments, offering an incremental improvement in unseen object instance segmentation.
The paper tackles the problem of segmenting unseen objects in real-world scenarios by addressing the sim2real domain shift, proposing a test-time adaptation method that improves segmentation results with state-of-the-art performance on overlap and boundary metrics.
Segmenting unseen objects is a crucial ability for the robot since it may encounter new environments during the operation. Recently, a popular solution is leveraging RGB-D features of large-scale synthetic data and directly applying the model to unseen real-world scenarios. However, the domain shift caused by the sim2real gap is inevitable, posing a crucial challenge to the segmentation model. In this paper, we emphasize the adaptation process across sim2real domains and model it as a learning problem on the BatchNorm parameters of a simulation-trained model. Specifically, we propose a novel non-parametric entropy objective, which formulates the learning objective for the test-time adaptation in an open-world manner. Then, a cross-modality knowledge distillation objective is further designed to encourage the test-time knowledge transfer for feature enhancement. Our approach can be efficiently implemented with only test images, without requiring annotations or revisiting the large-scale synthetic training data. Besides significant time savings, the proposed method consistently improves segmentation results on the overlap and boundary metrics, achieving state-of-the-art performance on unseen object instance segmentation.