A Closer Look at the Optimization Landscapes of Generative Adversarial Networks
This work addresses the training difficulties of GANs for researchers and practitioners in generative modeling, though it is incremental as it builds on existing theory with empirical validation.
The paper tackles the challenge of training Generative Adversarial Networks (GANs) by proposing new visualization techniques to study their optimization landscapes, showing empirically that training exhibits rotations around Local Stable Stationary Points and converges to saddle points for the generator loss while maintaining excellent performance.
Generative adversarial networks have been very successful in generative modeling, however they remain relatively challenging to train compared to standard deep neural networks. In this paper, we propose new visualization techniques for the optimization landscapes of GANs that enable us to study the game vector field resulting from the concatenation of the gradient of both players. Using these visualization techniques we try to bridge the gap between theory and practice by showing empirically that the training of GANs exhibits significant rotations around Local Stable Stationary Points (LSSP), similar to the one predicted by theory on toy examples. Moreover, we provide empirical evidence that GAN training converge to a stable stationary point which is a saddle point for the generator loss, not a minimum, while still achieving excellent performance.