LinkGAN: Linking GAN Latents to Pixels for Controllable Image Synthesis
This work addresses the need for more convenient local control in GAN generation for users in image synthesis applications, though it is incremental as it builds on existing GAN frameworks.
The paper tackles the problem of controllable image synthesis in GANs by introducing a regularizer that links latent space axes to specific pixels, enabling local control of image content through partial latent resampling. The result is a method that improves spatial controllability for both 2D and 3D-aware GAN models with minimal synthesis performance loss.
This work presents an easy-to-use regularizer for GAN training, which helps explicitly link some axes of the latent space to a set of pixels in the synthesized image. Establishing such a connection facilitates a more convenient local control of GAN generation, where users can alter the image content only within a spatial area simply by partially resampling the latent code. Experimental results confirm four appealing properties of our regularizer, which we call LinkGAN. (1) The latent-pixel linkage is applicable to either a fixed region (\textit{i.e.}, same for all instances) or a particular semantic category (i.e., varying across instances), like the sky. (2) Two or multiple regions can be independently linked to different latent axes, which further supports joint control. (3) Our regularizer can improve the spatial controllability of both 2D and 3D-aware GAN models, barely sacrificing the synthesis performance. (4) The models trained with our regularizer are compatible with GAN inversion techniques and maintain editability on real images.