3D dynamic hand gestures recognition using the Leap Motion sensor and convolutional neural networks
This work addresses gesture recognition for human-computer interaction in Virtual Reality, but it is incremental as it adapts existing methods to a specific sensor and dataset.
The paper tackles the problem of recognizing 3D dynamic hand gestures using a Leap Motion sensor by converting gesture data into color images and classifying them with a modified ResNet-50 CNN, achieving successful application on an existing dataset and preliminary real-time tests.
Defining methods for the automatic understanding of gestures is of paramount importance in many application contexts and in Virtual Reality applications for creating more natural and easy-to-use human-computer interaction methods. In this paper, we present a method for the recognition of a set of non-static gestures acquired through the Leap Motion sensor. The acquired gesture information is converted in color images, where the variation of hand joint positions during the gesture are projected on a plane and temporal information is represented with color intensity of the projected points. The classification of the gestures is performed using a deep Convolutional Neural Network (CNN). A modified version of the popular ResNet-50 architecture is adopted, obtained by removing the last fully connected layer and adding a new layer with as many neurons as the considered gesture classes. The method has been successfully applied to the existing reference dataset and preliminary tests have already been performed for the real-time recognition of dynamic gestures performed by users.