CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos
This work addresses the problem of data scarcity for generalist vision-language-action models in robotics by enabling effective skill transfer from human videos, representing an incremental advance with novel method components.
The paper tackles the challenge of transferring manipulation skills from abundant human videos to robotic systems by proposing CLAP, a framework that aligns visual and proprioceptive latent spaces using contrastive learning, resulting in significant performance improvements over baselines in robotic execution tasks.
Generalist Vision-Language-Action models are currently hindered by the scarcity of robotic data compared to the abundance of human video demonstrations. Existing Latent Action Models attempt to leverage video data but often suffer from visual entanglement, capturing noise rather than manipulation skills. To address this, we propose Contrastive Latent Action Pretraining (CLAP), a framework that aligns the visual latent space from videos with a proprioceptive latent space from robot trajectories. By employing contrastive learning, CLAP maps video transitions onto a quantized, physically executable codebook. Building on this representation, we introduce a dual-formulation VLA framework offering both CLAP-NTP, an autoregressive model excelling at instruction following and object generalization, and CLAP-RF, a Rectified Flow-based policy designed for high-frequency, precise manipulation. Furthermore, we propose a Knowledge Matching (KM) regularization strategy to mitigate catastrophic forgetting during fine-tuning. Extensive experiments demonstrate that CLAP significantly outperforms strong baselines, enabling the effective transfer of skills from human videos to robotic execution. Project page: https://lin-shan.com/CLAP/.