General In-Hand Object Rotation with Vision and Touch
This work addresses robotic manipulation challenges for tasks requiring precise object handling, though it appears incremental as it builds on existing multimodal sensing approaches.
The paper tackles the problem of in-hand object rotation using multimodal sensory inputs, achieving significant performance improvements over prior methods by leveraging a visuotactile transformer for online inference of object properties.
We introduce RotateIt, a system that enables fingertip-based object rotation along multiple axes by leveraging multimodal sensory inputs. Our system is trained in simulation, where it has access to ground-truth object shapes and physical properties. Then we distill it to operate on realistic yet noisy simulated visuotactile and proprioceptive sensory inputs. These multimodal inputs are fused via a visuotactile transformer, enabling online inference of object shapes and physical properties during deployment. We show significant performance improvements over prior methods and the importance of visual and tactile sensing.