Hand-Aware Egocentric Motion Reconstruction with Sequence-Level Context
This addresses the challenge of reliable 3D motion understanding for human-computer interaction in real-world egocentric vision systems, representing an incremental improvement over prior methods.
The paper tackles the problem of estimating full-body motion from first-person videos by introducing HaMoS, a hand-aware diffusion framework that conditions on head trajectories and intermittent hand cues, achieving state-of-the-art accuracy and temporal smoothness on public benchmarks.
Egocentric vision systems are becoming widely available, creating new opportunities for human-computer interaction. A core challenge is estimating the wearer's full-body motion from first-person videos, which is crucial for understanding human behavior. However, this task is difficult since most body parts are invisible from the egocentric view. Prior approaches mainly rely on head trajectories, leading to ambiguity, or assume continuously tracked hands, which is unrealistic for lightweight egocentric devices. In this work, we present HaMoS, the first hand-aware, sequence-level diffusion framework that directly conditions on both head trajectory and intermittently visible hand cues caused by field-of-view limitations and occlusions, as in real-world egocentric devices. To overcome the lack of datasets pairing diverse camera views with human motion, we introduce a novel augmentation method that models such real-world conditions. We also demonstrate that sequence-level contexts such as body shape and field-of-view are crucial for accurate motion reconstruction, and thus employ local attention to infer long sequences efficiently. Experiments on public benchmarks show that our method achieves state-of-the-art accuracy and temporal smoothness, demonstrating a practical step toward reliable in-the-wild egocentric 3D motion understanding.