On the Optimization Landscape of Maximum Mean Discrepancy
This provides theoretical guarantees for implicit generative models, addressing a key bottleneck in machine learning for researchers and practitioners, though it is incremental as it focuses on specific distributions.
The paper tackles the problem of understanding when generative models can globally optimize their non-convex objectives, specifically for Maximum Mean Discrepancy (MMD) learning, and proves that the MMD optimization landscape is benign for cases like Gaussian distributions with low-rank covariance and mixtures of Gaussians, enabling gradient-based methods to achieve global minimization.
Generative models have been successfully used for generating realistic signals. Because the likelihood function is typically intractable in most of these models, the common practice is to use "implicit" models that avoid likelihood calculation. However, it is hard to obtain theoretical guarantees for such models. In particular, it is not understood when they can globally optimize their non-convex objectives. Here we provide such an analysis for the case of Maximum Mean Discrepancy (MMD) learning of generative models. We prove several optimality results, including for a Gaussian distribution with low rank covariance (where likelihood is inapplicable) and a mixture of Gaussians. Our analysis shows that that the MMD optimization landscape is benign in these cases, and therefore gradient based methods will globally minimize the MMD objective.