SEAL: Self-supervised Embodied Active Learning using Exploration and 3D Consistency
This work addresses the challenge of label-efficient adaptation for robot vision, though it is incremental as it builds on existing self-supervised and active learning methods.
The paper tackles the problem of adapting internet-trained perception models to robot vision without extra labels by introducing the SEAL framework, which uses self-supervised active exploration and 3D consistency to improve object detection and instance segmentation, and demonstrates enhanced performance in Object Goal Navigation.
In this paper, we explore how we can build upon the data and models of Internet images and use them to adapt to robot vision without requiring any extra labels. We present a framework called Self-supervised Embodied Active Learning (SEAL). It utilizes perception models trained on internet images to learn an active exploration policy. The observations gathered by this exploration policy are labelled using 3D consistency and used to improve the perception model. We build and utilize 3D semantic maps to learn both action and perception in a completely self-supervised manner. The semantic map is used to compute an intrinsic motivation reward for training the exploration policy and for labelling the agent observations using spatio-temporal 3D consistency and label propagation. We demonstrate that the SEAL framework can be used to close the action-perception loop: it improves object detection and instance segmentation performance of a pretrained perception model by just moving around in training environments and the improved perception model can be used to improve Object Goal Navigation.