Enhanced Touchable Projector-depth System with Deep Hand Pose Estimation
This work addresses touch detection challenges for users of interactive projection systems, offering incremental improvements in gesture recognition and multi-finger disambiguation.
The paper tackled the problem of robust touch detection in projector-depth systems by combining surface touch detection with deep hand pose estimation, achieving improved detection of on- and above-surface gestures and fingertip positions despite measurement noise.
Touchable projection with structured light range cameras is a prolific medium for large interaction surfaces, affording multiple simultaneous users and simple, cheap setup. However robust touch detection in such projector-depth systems is difficult to achieve due to measurement noise. We propose a novel combination of surface touch detection and a deep network for hand pose estimation, which aids in detecting both on- and above-surface hand gestures, disambiguating multiple touch fingers, as well as recovering fingertip positions in face of noisy input. We present the details of our GPU-accelerated system and an evaluation of its performance, as well as applications such as an enhanced virtual keyboard that utilizes the added features.