Rewriting Geometric Rules of a GAN
This addresses the need for novice users to create custom generative models without large datasets, though it is incremental as it builds on existing GAN fine-tuning approaches.
The paper tackles the problem of enabling users to create generative models that synthesize objects with geometric changes beyond the original data distribution, by allowing warping of a GAN through editing a few outputs and applying low-rank updates with latent space augmentation, resulting in advantages over recent fine-tuning methods as shown in empirical tests.
Deep generative models make visual content creation more accessible to novice users by automating the synthesis of diverse, realistic content based on a collected dataset. However, the current machine learning approaches miss a key element of the creative process -- the ability to synthesize things that go far beyond the data distribution and everyday experience. To begin to address this issue, we enable a user to "warp" a given model by editing just a handful of original model outputs with desired geometric changes. Our method applies a low-rank update to a single model layer to reconstruct edited examples. Furthermore, to combat overfitting, we propose a latent space augmentation method based on style-mixing. Our method allows a user to create a model that synthesizes endless objects with defined geometric changes, enabling the creation of a new generative model without the burden of curating a large-scale dataset. We also demonstrate that edited models can be composed to achieve aggregated effects, and we present an interactive interface to enable users to create new models through composition. Empirical measurements on multiple test cases suggest the advantage of our method against recent GAN fine-tuning methods. Finally, we showcase several applications using the edited models, including latent space interpolation and image editing.