Hierarchical Temporal Transformer for 3D Hand Pose Estimation and Action Recognition from Egocentric RGB Videos
This addresses the challenge of self-occlusion and ambiguity in dynamic hand motions for applications like VR/AR and robotics, but it is incremental as it builds on existing transformer methods.
The paper tackled the problem of 3D hand pose estimation and action recognition from egocentric RGB videos by developing a hierarchical transformer framework, achieving competitive results on FPHA and H2O benchmarks.
Understanding dynamic hand motions and actions from egocentric RGB videos is a fundamental yet challenging task due to self-occlusion and ambiguity. To address occlusion and ambiguity, we develop a transformer-based framework to exploit temporal information for robust estimation. Noticing the different temporal granularity of and the semantic correlation between hand pose estimation and action recognition, we build a network hierarchy with two cascaded transformer encoders, where the first one exploits the short-term temporal cue for hand pose estimation, and the latter aggregates per-frame pose and object information over a longer time span to recognize the action. Our approach achieves competitive results on two first-person hand action benchmarks, namely FPHA and H2O. Extensive ablation studies verify our design choices.