3D-FM GAN: Towards 3D-Controllable Face Manipulation
This work addresses the problem of 3D-controllable face manipulation for applications in computer vision and graphics, offering an incremental improvement over existing GAN-based approaches.
The paper tackles the challenge of manipulating existing face images with precise 3D control by proposing 3D-FM GAN, a conditional GAN framework that enables high-quality, identity-preserved face manipulation without post-training tuning, outperforming prior methods in editability, identity preservation, and photo-realism.
3D-controllable portrait synthesis has significantly advanced, thanks to breakthroughs in generative adversarial networks (GANs). However, it is still challenging to manipulate existing face images with precise 3D control. While concatenating GAN inversion and a 3D-aware, noise-to-image GAN is a straight-forward solution, it is inefficient and may lead to noticeable drop in editing quality. To fill this gap, we propose 3D-FM GAN, a novel conditional GAN framework designed specifically for 3D-controllable face manipulation, and does not require any tuning after the end-to-end learning phase. By carefully encoding both the input face image and a physically-based rendering of 3D edits into a StyleGAN's latent spaces, our image generator provides high-quality, identity-preserved, 3D-controllable face manipulation. To effectively learn such novel framework, we develop two essential training strategies and a novel multiplicative co-modulation architecture that improves significantly upon naive schemes. With extensive evaluations, we show that our method outperforms the prior arts on various tasks, with better editability, stronger identity preservation, and higher photo-realism. In addition, we demonstrate a better generalizability of our design on large pose editing and out-of-domain images.