Lossless Adaptation of Pretrained Vision Models For Robotic Manipulation
This addresses the challenge of adapting versatile pretrained models for specialized robotic tasks without sacrificing their original functionality, which is incremental but practical for robotics applications.
The paper tackles the problem of fine-tuning pretrained vision models for robotic manipulation, which improves performance but causes representational drift and loss of versatility; the result is a 'lossless adaptation' method that reduces the performance gap by 35% compared to full fine-tuning while preserving original capabilities.
Recent works have shown that large models pretrained on common visual learning tasks can provide useful representations for a wide range of specialized perception problems, as well as a variety of robotic manipulation tasks. While prior work on robotic manipulation has predominantly used frozen pretrained features, we demonstrate that in robotics this approach can fail to reach optimal performance, and that fine-tuning of the full model can lead to significantly better results. Unfortunately, fine-tuning disrupts the pretrained visual representation, and causes representational drift towards the fine-tuned task thus leading to a loss of the versatility of the original model. We introduce "lossless adaptation" to address this shortcoming of classical fine-tuning. We demonstrate that appropriate placement of our parameter efficient adapters can significantly reduce the performance gap between frozen pretrained representations and full end-to-end fine-tuning without changes to the original representation and thus preserving original capabilities of the pretrained model. We perform a comprehensive investigation across three major model architectures (ViTs, NFNets, and ResNets), supervised (ImageNet-1K classification) and self-supervised pretrained weights (CLIP, BYOL, Visual MAE) in 3 task domains and 35 individual tasks, and demonstrate that our claims are strongly validated in various settings.