The Numerics of GANs
This addresses a fundamental problem in machine learning for researchers and practitioners by providing a more stable training method for GANs, though it is incremental as it builds on existing game theory frameworks.
The paper tackles the convergence issues in training Generative Adversarial Networks (GANs) by analyzing the gradient vector field and identifying problematic eigenvalues, resulting in a new algorithm that improves convergence and works on notoriously hard-to-train GAN architectures.
In this paper, we analyze the numerics of common algorithms for training Generative Adversarial Networks (GANs). Using the formalism of smooth two-player games we analyze the associated gradient vector field of GAN training objectives. Our findings suggest that the convergence of current algorithms suffers due to two factors: i) presence of eigenvalues of the Jacobian of the gradient vector field with zero real-part, and ii) eigenvalues with big imaginary part. Using these findings, we design a new algorithm that overcomes some of these limitations and has better convergence properties. Experimentally, we demonstrate its superiority on training common GAN architectures and show convergence on GAN architectures that are known to be notoriously hard to train.