Deep Multimodal Embedding: Manipulating Novel Objects with Point-clouds, Language and Trajectories
This work addresses the problem of multimodal reasoning for robots in real-world environments, offering a novel method for integrating vision, language, and motion data, though it is incremental in its approach to embedding learning.
The paper tackles the challenge of manually designing features for disparate sensor modalities in robotics by introducing a deep neural network algorithm that learns a shared embedding space for point-cloud, language, and trajectory data, achieving significant improvements in accuracy and inference time on a large dataset for manipulating novel objects.
A robot operating in a real-world environment needs to perform reasoning over a variety of sensor modalities such as vision, language and motion trajectories. However, it is extremely challenging to manually design features relating such disparate modalities. In this work, we introduce an algorithm that learns to embed point-cloud, natural language, and manipulation trajectory data into a shared embedding space with a deep neural network. To learn semantically meaningful spaces throughout our network, we use a loss-based margin to bring embeddings of relevant pairs closer together while driving less-relevant cases from different modalities further apart. We use this both to pre-train its lower layers and fine-tune our final embedding space, leading to a more robust representation. We test our algorithm on the task of manipulating novel objects and appliances based on prior experience with other objects. On a large dataset, we achieve significant improvements in both accuracy and inference time over the previous state of the art. We also perform end-to-end experiments on a PR2 robot utilizing our learned embedding space.