Robotic-CLIP: Fine-tuning CLIP on Action Data for Robotic Applications
This work addresses the need for improved robotic perception in dynamic action contexts, though it is incremental as it builds on the existing CLIP framework.
The paper tackled the problem of adapting CLIP, a vision-language model trained on static images, for robotic tasks involving dynamic actions by fine-tuning it on a large-scale action dataset of 309,433 videos (~7.4 million frames). The result is Robotic-CLIP, which outperforms other CLIP-based models in language-driven robotic tasks and demonstrates effectiveness in real-world grasping applications.
Vision language models have played a key role in extracting meaningful features for various robotic applications. Among these, Contrastive Language-Image Pretraining (CLIP) is widely used in robotic tasks that require both vision and natural language understanding. However, CLIP was trained solely on static images paired with text prompts and has not yet been fully adapted for robotic tasks involving dynamic actions. In this paper, we introduce Robotic-CLIP to enhance robotic perception capabilities. We first gather and label large-scale action data, and then build our Robotic-CLIP by fine-tuning CLIP on 309,433 videos (~7.4 million frames) of action data using contrastive learning. By leveraging action data, Robotic-CLIP inherits CLIP's strong image performance while gaining the ability to understand actions in robotic contexts. Intensive experiments show that our Robotic-CLIP outperforms other CLIP-based models across various language-driven robotic tasks. Additionally, we demonstrate the practical effectiveness of Robotic-CLIP in real-world grasping applications.