HpEIS: Learning Hand Pose Embeddings for Multimedia Interactive Systems
This work addresses usability issues in hand-pose-based interactive systems for multimedia applications, though it is incremental with improvements focused on stability and smoothing.
The authors tackled the problem of enabling stable and smooth mid-air hand movement interactions for multimedia systems by developing HpEIS, a hand-pose embedding system using a VAE, which reduced task completion time and distance to targets in experiments with 12 users.
We present a novel Hand-pose Embedding Interactive System (HpEIS) as a virtual sensor, which maps users' flexible hand poses to a two-dimensional visual space using a Variational Autoencoder (VAE) trained on a variety of hand poses. HpEIS enables visually interpretable and guidable support for user explorations in multimedia collections, using only a camera as an external hand pose acquisition device. We identify general usability issues associated with system stability and smoothing requirements through pilot experiments with expert and inexperienced users. We then design stability and smoothing improvements, including hand-pose data augmentation, an anti-jitter regularisation term added to loss function, stabilising post-processing for movement turning points and smoothing post-processing based on One Euro Filters. In target selection experiments (n=12), we evaluate HpEIS by measures of task completion time and the final distance to target points, with and without the gesture guidance window condition. Experimental responses indicate that HpEIS provides users with a learnable, flexible, stable and smooth mid-air hand movement interaction experience.