Making Method of Moments Great Again? -- How can GANs learn distributions
This provides theoretical insights into GAN training dynamics, addressing a central question in generative modeling for machine learning researchers, though it is incremental as it builds on existing GAN frameworks.
The paper tackles the problem of understanding when and how GANs learn target distributions by showing that early training forces the generator to match low-degree moments, and proves that matching these moments over polynomially many examples can learn distributions generated by two-layer neural networks.
Generative Adversarial Networks (GANs) are widely used models to learn complex real-world distributions. In GANs, the training of the generator usually stops when the discriminator can no longer distinguish the generator's output from the set of training examples. A central question of GANs is that when the training stops, whether the generated distribution is actually close to the target distribution, and how the training process reaches to such configurations efficiently? In this paper, we established a theoretical results towards understanding this generator-discriminator training process. We empirically observe that during the earlier stage of the GANs training, the discriminator is trying to force the generator to match the low degree moments between the generator's output and the target distribution. Moreover, only by matching these empirical moments over polynomially many training examples, we prove that the generator can already learn notable class of distributions, including those that can be generated by two-layer neural networks.