Dynamic Hand Gesture-Featured Human Motor Adaptation in Tool Delivery using Voice Recognition
This addresses the need for user-friendly interfaces in human-robot collaboration, particularly for tasks like soldering where users work with both hands, though it is incremental as it combines existing recognition methods.
The paper tackled the problem of intuitive human-robot collaboration by developing a multimodal framework integrating hand gesture, dynamic movement, and voice recognition for tool delivery, achieving 94.3% accuracy in static hand gesture recognition and 97.6% in dynamic motion recognition.
Human-robot collaboration has benefited users with higher efficiency towards interactive tasks. Nevertheless, most collaborative schemes rely on complicated human-machine interfaces, which might lack the requisite intuitiveness compared with natural limb control. We also expect to understand human intent with low training data requirements. In response to these challenges, this paper introduces an innovative human-robot collaborative framework that seamlessly integrates hand gesture and dynamic movement recognition, voice recognition, and a switchable control adaptation strategy. These modules provide a user-friendly approach that enables the robot to deliver the tools as per user need, especially when the user is working with both hands. Therefore, users can focus on their task execution without additional training in the use of human-machine interfaces, while the robot interprets their intuitive gestures. The proposed multimodal interaction framework is executed in the UR5e robot platform equipped with a RealSense D435i camera, and the effectiveness is assessed through a soldering circuit board task. The experiment results have demonstrated superior performance in hand gesture recognition, where the static hand gesture recognition module achieves an accuracy of 94.3\%, while the dynamic motion recognition module reaches 97.6\% accuracy. Compared with human solo manipulation, the proposed approach facilitates higher efficiency tool delivery, without significantly distracting from human intents.