MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamics
This work addresses the problem of multimodal human motion generation for applications in animation and robotics, but it appears incremental as it builds on existing VAE frameworks for motion modeling.
The paper tackled generating diverse and plausible long-term human motion sequences by modeling them as transitions between motion modes, using a Motion Transformation VAE (MT-VAE) that learns embeddings and transformations for these modes, and demonstrated applications like motion transfer and video synthesis on facial and full-body data.
Long-term human motion can be represented as a series of motion modes---motion sequences that capture short-term temporal dynamics---with transitions between them. We leverage this structure and present a novel Motion Transformation Variational Auto-Encoders (MT-VAE) for learning motion sequence generation. Our model jointly learns a feature embedding for motion modes (that the motion sequence can be reconstructed from) and a feature transformation that represents the transition of one motion mode to the next motion mode. Our model is able to generate multiple diverse and plausible motion sequences in the future from the same input. We apply our approach to both facial and full body motion, and demonstrate applications like analogy-based motion transfer and video synthesis.