GAT-GMM: Generative Adversarial Training for Gaussian Mixture Models
This work addresses a specific bottleneck in generative modeling for multi-modal distributions, offering an incremental improvement for researchers in machine learning and statistics.
The paper tackles the problem of learning Gaussian mixture models (GMMs) using generative adversarial networks (GANs), which perform suboptimally on such multi-modal distributions. It proposes GAT-GMM, a minimax GAN framework that, in experiments, performs as well as the expectation-maximization algorithm for mixtures of two Gaussians.
Generative adversarial networks (GANs) learn the distribution of observed samples through a zero-sum game between two machine players, a generator and a discriminator. While GANs achieve great success in learning the complex distribution of image, sound, and text data, they perform suboptimally in learning multi-modal distribution-learning benchmarks including Gaussian mixture models (GMMs). In this paper, we propose Generative Adversarial Training for Gaussian Mixture Models (GAT-GMM), a minimax GAN framework for learning GMMs. Motivated by optimal transport theory, we design the zero-sum game in GAT-GMM using a random linear generator and a softmax-based quadratic discriminator architecture, which leads to a non-convex concave minimax optimization problem. We show that a Gradient Descent Ascent (GDA) method converges to an approximate stationary minimax point of the GAT-GMM optimization problem. In the benchmark case of a mixture of two symmetric, well-separated Gaussians, we further show this stationary point recovers the true parameters of the underlying GMM. We numerically support our theoretical findings by performing several experiments, which demonstrate that GAT-GMM can perform as well as the expectation-maximization algorithm in learning mixtures of two Gaussians.