Can Action be Imitated? Learn to Reconstruct and Transfer Human Dynamics from Videos
This work addresses the challenge of action imitation from videos for applications in animation and robotics, presenting a novel task and method.
The paper tackles the problem of imitating human actions from videos by reconstructing and transferring human dynamics to a target mesh, achieving high-quality and detailed human body mesh generation as demonstrated in experiments.
Given a video demonstration, can we imitate the action contained in this video? In this paper, we introduce a novel task, dubbed mesh-based action imitation. The goal of this task is to enable an arbitrary target human mesh to perform the same action shown on the video demonstration. To achieve this, a novel Mesh-based Video Action Imitation (M-VAI) method is proposed by us. M-VAI first learns to reconstruct the meshes from the given source image frames, then the initial recovered mesh sequence is fed into mesh2mesh, a mesh sequence smooth module proposed by us, to improve the temporal consistency. Finally, we imitate the actions by transferring the pose from the constructed human body to our target identity mesh. High-quality and detailed human body meshes can be generated by using our M-VAI. Extensive experiments demonstrate the feasibility of our task and the effectiveness of our proposed method.