Leveraging RGB Images for Pre-Training of Event-Based Hand Pose Estimation
This addresses a data bottleneck for researchers and practitioners in event-based vision, enabling more accurate hand pose estimation with minimal labeled samples, though it is an incremental advance in pseudo-event generation.
The paper tackles the scarcity of labeled training data for event-based 3D hand pose estimation by introducing RPEP, a pre-training method that uses labeled RGB images and unlabeled event data to generate realistic pseudo-events for moving hands, resulting in up to 24% improvement on the EvRealHands benchmark.
This paper presents RPEP, the first pre-training method for event-based 3D hand pose estimation using labeled RGB images and unpaired, unlabeled event data. Event data offer significant benefits such as high temporal resolution and low latency, but their application to hand pose estimation is still limited by the scarcity of labeled training data. To address this, we repurpose real RGB datasets to train event-based estimators. This is done by constructing pseudo-event-RGB pairs, where event data is generated and aligned with the ground-truth poses of RGB images. Unfortunately, existing pseudo-event generation techniques assume stationary objects, thus struggling to handle non-stationary, dynamically moving hands. To overcome this, RPEP introduces a novel generation strategy that decomposes hand movements into smaller, step-by-step motions. This decomposition allows our method to capture temporal changes in articulation, constructing more realistic event data for a moving hand. Additionally, RPEP imposes a motion reversal constraint, regularizing event generation using reversed motion. Extensive experiments show that our pre-trained model significantly outperforms state-of-the-art methods on real event data, achieving up to 24% improvement on EvRealHands. Moreover, it delivers strong performance with minimal labeled samples for fine-tuning, making it well-suited for practical deployment.