Exploring Visual Pre-training for Robot Manipulation: Datasets, Models and Methods
This work addresses the problem of optimizing visual pre-training for robot manipulation tasks, providing incremental improvements through systematic analysis and a new scheme.
The paper investigates visual pre-training strategies for robot manipulation, proposing Vi-PRoM, a scheme combining self-supervised and supervised learning, which demonstrates superiority in experiments across simulation and real-world environments.
Visual pre-training with large-scale real-world data has made great progress in recent years, showing great potential in robot learning with pixel observations. However, the recipes of visual pre-training for robot manipulation tasks are yet to be built. In this paper, we thoroughly investigate the effects of visual pre-training strategies on robot manipulation tasks from three fundamental perspectives: pre-training datasets, model architectures and training methods. Several significant experimental findings are provided that are beneficial for robot learning. Further, we propose a visual pre-training scheme for robot manipulation termed Vi-PRoM, which combines self-supervised learning and supervised learning. Concretely, the former employs contrastive learning to acquire underlying patterns from large-scale unlabeled data, while the latter aims learning visual semantics and temporal dynamics. Extensive experiments on robot manipulations in various simulation environments and the real robot demonstrate the superiority of the proposed scheme. Videos and more details can be found on \url{https://explore-pretrain-robot.github.io}.