On the Discrimination-Generalization Tradeoff in GANs
This work addresses the fundamental challenge of balancing discrimination and generalization in GAN training for machine learning practitioners, offering theoretical insights into practical performance, though it is incremental as it builds on existing GAN analysis.
The paper tackles the tradeoff between discriminative power and generalization in GANs by showing that a discriminator set is discriminative if its linear span is dense in bounded continuous functions, a condition met even by simple neural networks, and provides generalization bounds under metrics like neural distance and KL divergence, with bounds showing generalization is guaranteed for small discriminator sets regardless of generator size.
Generative adversarial training can be generally understood as minimizing certain moment matching loss defined by a set of discriminator functions, typically neural networks. The discriminator set should be large enough to be able to uniquely identify the true distribution (discriminative), and also be small enough to go beyond memorizing samples (generalizable). In this paper, we show that a discriminator set is guaranteed to be discriminative whenever its linear span is dense in the set of bounded continuous functions. This is a very mild condition satisfied even by neural networks with a single neuron. Further, we develop generalization bounds between the learned distribution and true distribution under different evaluation metrics. When evaluated with neural distance, our bounds show that generalization is guaranteed as long as the discriminator set is small enough, regardless of the size of the generator or hypothesis set. When evaluated with KL divergence, our bound provides an explanation on the counter-intuitive behaviors of testing likelihood in GAN training. Our analysis sheds lights on understanding the practical performance of GANs.