ViMo: Generating Motions from Casual Videos
This work addresses the problem of limited motion data for computer vision researchers by enabling scalable motion generation from casual videos, though it is incremental as it builds on diffusion models.
The paper tackles the challenge of generating 3D human motions from casual videos with complex camera movements and occlusions, proposing ViMo, a framework that leverages untapped video content to produce diverse and realistic motions, as demonstrated by experimental results showing natural motion generation even in difficult scenarios.
Although humans have the innate ability to imagine multiple possible actions from videos, it remains an extraordinary challenge for computers due to the intricate camera movements and montages. Most existing motion generation methods predominantly rely on manually collected motion datasets, usually tediously sourced from motion capture (Mocap) systems or Multi-View cameras, unavoidably resulting in a limited size that severely undermines their generalizability. Inspired by recent advance of diffusion models, we probe a simple and effective way to capture motions from videos and propose a novel Video-to-Motion-Generation framework (ViMo) which could leverage the immense trove of untapped video content to produce abundant and diverse 3D human motions. Distinct from prior work, our videos could be more causal, including complicated camera movements and occlusions. Striking experimental results demonstrate the proposed model could generate natural motions even for videos where rapid movements, varying perspectives, or frequent occlusions might exist. We also show this work could enable three important downstream applications, such as generating dancing motions according to arbitrary music and source video style. Extensive experimental results prove that our model offers an effective and scalable way to generate diversity and realistic motions. Code and demos will be public soon.