Cross-identity Video Motion Retargeting with Joint Transformation and Synthesis
This addresses the problem of generating realistic cross-identity videos for applications like animation or video editing, though it appears incremental as it builds on existing retargeting methods with a novel dual-branch design.
The paper tackles video motion retargeting by proposing TS-Net, a dual-branch network that combines warp-based transformation and warp-free synthesis to generate videos with subject appearance from one video and motion from another, achieving better performance than state-of-the-art models on face and dance datasets.
In this paper, we propose a novel dual-branch Transformation-Synthesis network (TS-Net), for video motion retargeting. Given one subject video and one driving video, TS-Net can produce a new plausible video with the subject appearance of the subject video and motion pattern of the driving video. TS-Net consists of a warp-based transformation branch and a warp-free synthesis branch. The novel design of dual branches combines the strengths of deformation-grid-based transformation and warp-free generation for better identity preservation and robustness to occlusion in the synthesized videos. A mask-aware similarity module is further introduced to the transformation branch to reduce computational overhead. Experimental results on face and dance datasets show that TS-Net achieves better performance in video motion retargeting than several state-of-the-art models as well as its single-branch variants. Our code is available at https://github.com/nihaomiao/WACV23_TSNet.