Visuo-Tactile Transformers for Manipulation
This work addresses manipulation challenges for robotics by integrating vision and touch, though it appears incremental as it extends existing transformer methods to multimodal feedback.
The paper tackled the problem of improving robot manipulation by learning joint visuo-tactile representations, resulting in enhanced dexterity and sample efficiency, with evaluations showing efficacy in simulated and real-world tasks.
Learning representations in the joint domain of vision and touch can improve manipulation dexterity, robustness, and sample-complexity by exploiting mutual information and complementary cues. Here, we present Visuo-Tactile Transformers (VTTs), a novel multimodal representation learning approach suited for model-based reinforcement learning and planning. Our approach extends the Visual Transformer \cite{dosovitskiy2021image} to handle visuo-tactile feedback. Specifically, VTT uses tactile feedback together with self and cross-modal attention to build latent heatmap representations that focus attention on important task features in the visual domain. We demonstrate the efficacy of VTT for representation learning with a comparative evaluation against baselines on four simulated robot tasks and one real world block pushing task. We conduct an ablation study over the components of VTT to highlight the importance of cross-modality in representation learning.