You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions
This addresses the challenge of inferring body pose in applications like augmented reality and healthcare where much of the wearer's body is out of view, representing a novel approach but incremental in its domain-specific focus.
The paper tackles the problem of estimating a camera wearer's 3D body pose from egocentric video by leveraging interactions with another person whose pose is visible, showing substantial improvements in state-of-the-art egocentric body pose estimation.
The body pose of a person wearing a camera is of great interest for applications in augmented reality, healthcare, and robotics, yet much of the person's body is out of view for a typical wearable camera. We propose a learning-based approach to estimate the camera wearer's 3D body pose from egocentric video sequences. Our key insight is to leverage interactions with another person---whose body pose we can directly observe---as a signal inherently linked to the body pose of the first-person subject. We show that since interactions between individuals often induce a well-ordered series of back-and-forth responses, it is possible to learn a temporal model of the interlinked poses even though one party is largely out of view. We demonstrate our idea on a variety of domains with dyadic interaction and show the substantial impact on egocentric body pose estimation, which improves the state of the art. Video results are available at http://vision.cs.utexas.edu/projects/you2me/