The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning
This work addresses the challenge of robust and generalizable robotic manipulation for applications in automation and robotics, representing an incremental advance by integrating multimodal learning into reinforcement learning.
The paper tackled the problem of fusing visual and tactile information for robotic manipulation tasks by proposing Masked Multimodal Learning (M3L), which improved sample efficiency and enabled generalization beyond single-sense policies, as demonstrated in simulated environments like insertion, door opening, and in-hand manipulation.
Humans rely on the synergy of their senses for most essential tasks. For tasks requiring object manipulation, we seamlessly and effectively exploit the complementarity of our senses of vision and touch. This paper draws inspiration from such capabilities and aims to find a systematic approach to fuse visual and tactile information in a reinforcement learning setting. We propose Masked Multimodal Learning (M3L), which jointly learns a policy and visual-tactile representations based on masked autoencoding. The representations jointly learned from vision and touch improve sample efficiency, and unlock generalization capabilities beyond those achievable through each of the senses separately. Remarkably, representations learned in a multimodal setting also benefit vision-only policies at test time. We evaluate M3L on three simulated environments with both visual and tactile observations: robotic insertion, door opening, and dexterous in-hand manipulation, demonstrating the benefits of learning a multimodal policy. Code and videos of the experiments are available at https://sferrazza.cc/m3l_site.