3D Magic Mirror: Automatic Video to 3D Caricature Translation
This addresses the need for efficient 3D caricature creation for entertainment or digital art applications, representing an incremental advance by combining existing techniques like VAE-CycleGAN and CariGANs.
The paper tackles the problem of automatically generating expressive 3D caricatures from 2D photos or videos in real-time, achieving results that meet people's expectations for 3D caricature style as shown in a perceptual study.
Caricature is an abstraction of a real person which distorts or exaggerates certain features, but still retains a likeness. While most existing works focus on 3D caricature reconstruction from 2D caricatures or translating 2D photos to 2D caricatures, this paper presents a real-time and automatic algorithm for creating expressive 3D caricatures with caricature style texture map from 2D photos or videos. To solve this challenging ill-posed reconstruction problem and cross-domain translation problem, we first reconstruct the 3D face shape for each frame, and then translate 3D face shape from normal style to caricature style by a novel identity and expression preserving VAE-CycleGAN. Based on a labeling formulation, the caricature texture map is constructed from a set of multi-view caricature images generated by CariGANs. The effectiveness and efficiency of our method are demonstrated by comparison with baseline implementations. The perceptual study shows that the 3D caricatures generated by our method meet people's expectations of 3D caricature style.