Deep Gated Multi-modal Learning: In-hand Object Pose Changes Estimation using Tactile and Image Data
This addresses the challenge of occlusion and sensor noise in robotic in-hand manipulation, offering an incremental improvement over existing multi-modal methods.
The paper tackled the problem of estimating in-hand object pose changes during manipulation by proposing a deep gated multi-modal learning method that self-determines reliability values for tactile and image data, achieving estimation even for unknown objects with 15 objects tested.
For in-hand manipulation, estimation of the object pose inside the hand is one of the important functions to manipulate objects to the target pose. Since in-hand manipulation tends to cause occlusions by the hand or the object itself, image information only is not sufficient for in-hand object pose estimation. Multiple modalities can be used in this case, the advantage is that other modalities can compensate for occlusion, noise, and sensor malfunctions. Even though deciding the utilization rate of a modality (referred to as reliability value) corresponding to the situations is important, the manual design of such models is difficult, especially for various situations. In this paper, we propose deep gated multi-modal learning, which self-determines the reliability value of each modality through end-to-end deep learning. For the experiments, an RGB camera and a GelSight tactile sensor were attached to the parallel gripper of the Sawyer robot, and the object pose changes were estimated during grasping. A total of 15 objects were used in the experiments. In the proposed model, the reliability values of the modalities were determined according to the noise level and failure of each modality, and it was confirmed that the pose change was estimated even for unknown objects.