On the "steerability" of generative adversarial networks
This work addresses the problem of biased generalization in generative models for computer vision researchers, highlighting incremental improvements in understanding GAN limitations.
The paper investigates the limited ability of current generative adversarial networks (GANs) to generalize beyond training data biases, such as centered objects, by testing their capacity to fit simple transformations like camera movements and color changes, and finds that while GANs can be 'steered' in latent space to shift distributions and create realistic images, the degree of shift depends on training data breadth, with experiments quantifying these limits and proposing mitigation techniques.
An open secret in contemporary machine learning is that many models work beautifully on standard benchmarks but fail to generalize outside the lab. This has been attributed to biased training data, which provide poor coverage over real world events. Generative models are no exception, but recent advances in generative adversarial networks (GANs) suggest otherwise - these models can now synthesize strikingly realistic and diverse images. Is generative modeling of photos a solved problem? We show that although current GANs can fit standard datasets very well, they still fall short of being comprehensive models of the visual manifold. In particular, we study their ability to fit simple transformations such as camera movements and color changes. We find that the models reflect the biases of the datasets on which they are trained (e.g., centered objects), but that they also exhibit some capacity for generalization: by "steering" in latent space, we can shift the distribution while still creating realistic images. We hypothesize that the degree of distributional shift is related to the breadth of the training data distribution. Thus, we conduct experiments to quantify the limits of GAN transformations and introduce techniques to mitigate the problem. Code is released on our project page: https://ali-design.github.io/gan_steerability/