Combining Vision and EMG-Based Hand Tracking for Extended Reality Musical Instruments
This addresses hand tracking challenges for XR users, but it is incremental as it builds on existing multimodal methods.
The paper tackled the problem of self-occlusion in vision-based hand tracking for extended reality musical instruments by combining vision with surface electromyography data, resulting in significantly improved tracking accuracy for finger joints prone to occlusion.
Hand tracking is a critical component of natural user interactions in extended reality (XR) environments, including extended reality musical instruments (XRMIs). However, self-occlusion remains a significant challenge for vision-based hand tracking systems, leading to inaccurate results and degraded user experiences. In this paper, we propose a multimodal hand tracking system that combines vision-based hand tracking with surface electromyography (sEMG) data for finger joint angle estimation. We validate the effectiveness of our system through a series of hand pose tasks designed to cover a wide range of gestures, including those prone to self-occlusion. By comparing the performance of our multimodal system to a baseline vision-based tracking method, we demonstrate that our multimodal approach significantly improves tracking accuracy for several finger joints prone to self-occlusion. These findings suggest that our system has the potential to enhance XR experiences by providing more accurate and robust hand tracking, even in the presence of self-occlusion.