On Duality Gap as a Measure for Monitoring GAN Training
This work provides a more reliable metric for researchers and practitioners to monitor and tune Generative Adversarial Networks, which are notoriously difficult to train.
The paper addresses the limitations of the duality gap as a measure for monitoring GAN training, which stems from its estimation process. It proposes a more dependable estimation method based on local perturbations, demonstrating its efficacy in capturing GAN training progress and identifying model convergence/divergence with minimal computational overhead.
Generative adversarial network (GAN) is among the most popular deep learning models for learning complex data distributions. However, training a GAN is known to be a challenging task. This is often attributed to the lack of correlation between the training progress and the trajectory of the generator and discriminator losses and the need for the GAN's subjective evaluation. A recently proposed measure inspired by game theory - the duality gap, aims to bridge this gap. However, as we demonstrate, the duality gap's capability remains constrained due to limitations posed by its estimation process. This paper presents a theoretical understanding of this limitation and proposes a more dependable estimation process for the duality gap. At the crux of our approach is the idea that local perturbations can help agents in a zero-sum game escape non-Nash saddle points efficiently. Through exhaustive experimentation across GAN models and datasets, we establish the efficacy of our approach in capturing the GAN training progress with minimal increase to the computational complexity. Further, we show that our estimate, with its ability to identify model convergence/divergence, is a potential performance measure that can be used to tune the hyperparameters of a GAN.