Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks
This addresses the challenge of sample-efficient robot control in unstructured environments, though it is incremental as it builds on existing self-supervised and reinforcement learning methods.
The paper tackled the problem of learning control policies for contact-rich manipulation tasks by using self-supervised learning to create a compact multimodal representation from vision and touch inputs, which improved sample efficiency in peg insertion tasks across various conditions and perturbations.
Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. However, it is non-trivial to manually design a robot controller that combines modalities with very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to deploy on real robots due to sample complexity. We use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample efficiency of our policy learning. We evaluate our method on a peg insertion task, generalizing over different geometry, configurations, and clearances, while being robust to external perturbations. Results for simulated and real robot experiments are presented.