Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamics
This addresses the challenge of realistic hand gesture synthesis and estimation for applications in human-computer interaction and animation, representing a novel method for a known bottleneck.
The paper tackles the problem of inferring 3D hand shapes from body motion in conversational gestures, achieving state-of-the-art performance and generalization to multi-person conversations.
We propose a novel learned deep prior of body motion for 3D hand shape synthesis and estimation in the domain of conversational gestures. Our model builds upon the insight that body motion and hand gestures are strongly correlated in non-verbal communication settings. We formulate the learning of this prior as a prediction task of 3D hand shape over time given body motion input alone. Trained with 3D pose estimations obtained from a large-scale dataset of internet videos, our hand prediction model produces convincing 3D hand gestures given only the 3D motion of the speaker's arms as input. We demonstrate the efficacy of our method on hand gesture synthesis from body motion input, and as a strong body prior for single-view image-based 3D hand pose estimation. We demonstrate that our method outperforms previous state-of-the-art approaches and can generalize beyond the monologue-based training data to multi-person conversations. Video results are available at http://people.eecs.berkeley.edu/~evonne_ng/projects/body2hands/.