3D Pose Estimation of Two Interacting Hands from a Monocular Event Camera
It solves the problem of accurate 3D hand pose estimation under challenging conditions like low light and fast motion for applications in robotics and VR, representing a novel domain-specific advancement.
The paper tackles 3D tracking of two fast-moving, interacting hands from a monocular event camera, introducing a framework that addresses left-right hand ambiguity and hand collisions, and outperforms existing methods in 3D reconstruction accuracy while generalizing to real data under severe light conditions.
3D hand tracking from a monocular video is a very challenging problem due to hand interactions, occlusions, left-right hand ambiguity, and fast motion. Most existing methods rely on RGB inputs, which have severe limitations under low-light conditions and suffer from motion blur. In contrast, event cameras capture local brightness changes instead of full image frames and do not suffer from the described effects. Unfortunately, existing image-based techniques cannot be directly applied to events due to significant differences in the data modalities. In response to these challenges, this paper introduces the first framework for 3D tracking of two fast-moving and interacting hands from a single monocular event camera. Our approach tackles the left-right hand ambiguity with a novel semi-supervised feature-wise attention mechanism and integrates an intersection loss to fix hand collisions. To facilitate advances in this research domain, we release a new synthetic large-scale dataset of two interacting hands, Ev2Hands-S, and a new real benchmark with real event streams and ground-truth 3D annotations, Ev2Hands-R. Our approach outperforms existing methods in terms of the 3D reconstruction accuracy and generalises to real data under severe light conditions.