Collaborative Representation Learning for Alignment of Tactile, Language, and Vision Modalities
This work addresses the challenge of redundant features and lack of integration in multimodal tactile sensing for robotics, though it appears incremental as it builds on CLIP-based methods.
The paper tackles the problem of standardizing tactile sensing and integrating it with vision and language for robots by proposing TLV-CoRe, a method that improves sensor-agnostic representation learning and cross-modal alignment, as demonstrated through experimental results.
Tactile sensing offers rich and complementary information to vision and language, enabling robots to perceive fine-grained object properties. However, existing tactile sensors lack standardization, leading to redundant features that hinder cross-sensor generalization. Moreover, existing methods fail to fully integrate the intermediate communication among tactile, language, and vision modalities. To address this, we propose TLV-CoRe, a CLIP-based Tactile-Language-Vision Collaborative Representation learning method. TLV-CoRe introduces a Sensor-Aware Modulator to unify tactile features across different sensors and employs tactile-irrelevant decoupled learning to disentangle irrelevant tactile features. Additionally, a Unified Bridging Adapter is introduced to enhance tri-modal interaction within the shared representation space. To fairly evaluate the effectiveness of tactile models, we further propose the RSS evaluation framework, focusing on Robustness, Synergy, and Stability across different methods. Experimental results demonstrate that TLV-CoRe significantly improves sensor-agnostic representation learning and cross-modal alignment, offering a new direction for multimodal tactile representation.