On-device Real-time Hand Gesture Recognition
This work addresses on-device gesture recognition for applications like human-computer interaction, but it is incremental as it builds on existing methods like MediaPipe Hands.
The authors tackled real-time hand gesture recognition on-device by improving a hand skeleton tracker for better keypoint accuracy and 3D estimation, and developed two classifiers (heuristic and neural network) to detect predefined static gestures from RGB camera input.
We present an on-device real-time hand gesture recognition (HGR) system, which detects a set of predefined static gestures from a single RGB camera. The system consists of two parts: a hand skeleton tracker and a gesture classifier. We use MediaPipe Hands as the basis of the hand skeleton tracker, improve the keypoint accuracy, and add the estimation of 3D keypoints in a world metric space. We create two different gesture classifiers, one based on heuristics and the other using neural networks (NN).