On the Limitations of First-Order Approximation in GAN Dynamics
This addresses the problem of unstable GAN training for machine learning practitioners, providing theoretical insights into a known bottleneck, though it is incremental as it builds on existing GAN analysis.
The paper tackles the problem of understanding GAN learning dynamics by analyzing a parametric model that exhibits issues like vanishing gradients and mode collapse. It finds that while GANs with optimal discriminators converge, first-order approximations lead to unstable dynamics and mode collapse, suggesting this contributes to training challenges.
While Generative Adversarial Networks (GANs) have demonstrated promising performance on multiple vision tasks, their learning dynamics are not yet well understood, both in theory and in practice. To address this issue, we study GAN dynamics in a simple yet rich parametric model that exhibits several of the common problematic convergence behaviors such as vanishing gradients, mode collapse, and diverging or oscillatory behavior. In spite of the non-convex nature of our model, we are able to perform a rigorous theoretical analysis of its convergence behavior. Our analysis reveals an interesting dichotomy: a GAN with an optimal discriminator provably converges, while first order approximations of the discriminator steps lead to unstable GAN dynamics and mode collapse. Our result suggests that using first order discriminator steps (the de-facto standard in most existing GAN setups) might be one of the factors that makes GAN training challenging in practice.