The Invisible EgoHand: 3D Hand Forecasting through EgoBody Pose Estimation
This addresses the limitation in egocentric vision for understanding human intention by enabling hand forecasting beyond visible fields, though it is incremental as it builds on existing datasets and methods.
The paper tackles the problem of forecasting 3D hand motion and pose from egocentric video, even when hands are out of view, by proposing a diffusion-based transformer method that leverages full-body pose constraints, resulting in improvements of 3.4cm in ADE for trajectory forecasting and 5.1cm in MPJPE for pose forecasting over baselines.
Forecasting hand motion and pose from an egocentric perspective is essential for understanding human intention. However, existing methods focus solely on predicting positions without considering articulation, and only when the hands are visible in the field of view. This limitation overlooks the fact that approximate hand positions can still be inferred even when they are outside the camera's view. In this paper, we propose a method to forecast the 3D trajectories and poses of both hands from an egocentric video, both in and out of the field of view. We propose a diffusion-based transformer architecture for Egocentric Hand Forecasting, EgoH4, which takes as input the observation sequence and camera poses, then predicts future 3D motion and poses for both hands of the camera wearer. We leverage full-body pose information, allowing other joints to provide constraints on hand motion. We denoise the hand and body joints along with a visibility predictor for hand joints and a 3D-to-2D reprojection loss that minimizes the error when hands are in-view. We evaluate EgoH4 on the Ego-Exo4D dataset, combining subsets with body and hand annotations. We train on 156K sequences and evaluate on 34K sequences, respectively. EgoH4 improves the performance by 3.4cm and 5.1cm over the baseline in terms of ADE for hand trajectory forecasting and MPJPE for hand pose forecasting. Project page: https://masashi-hatano.github.io/EgoH4/