SPIRIT: Perceptive Shared Autonomy for Robust Robotic Manipulation under Deep Learning Uncertainty
This work provides a method for safely integrating high-performing but uninterpretable deep learning methods into robotic systems, particularly for applications requiring robust manipulation despite perception uncertainties.
This paper addresses the challenge of deploying deep learning (DL) in safety-critical robotic manipulation by proposing perceptive shared autonomy. The system, SPIRIT, dynamically adjusts autonomy levels based on DL perception uncertainty, enabling semi-autonomous control when confident and transitioning to haptic teleoperation during high uncertainty. This approach improves both manipulation performance and system reliability, even when DL perception fails.
Deep learning (DL) has enabled impressive advances in robotic perception, yet its limited robustness and lack of interpretability hinder reliable deployment in safety critical applications. We propose a concept termed perceptive shared autonomy, in which uncertainty estimates from DL based perception are used to regulate the level of autonomy. Specifically, when the robot's perception is confident, semi-autonomous manipulation is enabled to improve performance; when uncertainty increases, control transitions to haptic teleoperation for maintaining robustness. In this way, high-performing but uninterpretable DL methods can be integrated safely into robotic systems. A key technical enabler is an uncertainty aware DL based point cloud registration approach based on the so called Neural Tangent Kernels (NTK). We evaluate perceptive shared autonomy on challenging aerial manipulation tasks through a user study of 15 participants and realization of mock-up industrial scenarios, demonstrating reliable robotic manipulation despite failures in DL based perception. The resulting system, named SPIRIT, improves both manipulation performance and system reliability. SPIRIT was selected as a finalist of a major industrial innovation award.