Controllable GAN Synthesis Using Non-Rigid Structure-from-Motion
This provides a versatile tool for modeling and editing geometry in faces, but it is incremental as it builds on existing NRSfM and GAN methods.
The paper tackles the problem of controlling 3D geometry in 2D GAN-generated images by combining non-rigid structure-from-motion with deep generative models, enabling editing of camera and shape parameters without retraining and synthesizing novel shapes from arbitrary views.
In this paper, we present an approach for combining non-rigid structure-from-motion (NRSfM) with deep generative models,and propose an efficient framework for discovering trajectories in the latent space of 2D GANs corresponding to changes in 3D geometry. Our approach uses recent advances in NRSfM and enables editing of the camera and non-rigid shape information associated with the latent codes without needing to retrain the generator. This formulation provides an implicit dense 3D reconstruction as it enables the image synthesis of novel shapes from arbitrary view angles and non-rigid structure. The method is built upon a sparse backbone, where a neural regressor is first trained to regress parameters describing the cameras and sparse non-rigid structure directly from the latent codes. The latent trajectories associated with changes in the camera and structure parameters are then identified by estimating the local inverse of the regressor in the neighborhood of a given latent code. The experiments show that our approach provides a versatile, systematic way to model, analyze, and edit the geometry and non-rigid structures of faces.