Sobolev GAN
This work addresses distribution comparison in GANs for machine learning practitioners, offering a novel metric that could improve training stability and performance in tasks like text generation and semi-supervised learning, but it appears incremental as an extension of existing IPM frameworks.
The authors tackled the problem of comparing high-dimensional distributions in generative adversarial networks (GANs) by proposing the Sobolev Integral Probability Metric (IPM), which extends one-dimensional statistical measures to high dimensions and enforces smoothness on the critic. They demonstrated competitive results in semi-supervised learning on CIFAR-10, though specific numerical gains were not detailed.
We propose a new Integral Probability Metric (IPM) between distributions: the Sobolev IPM. The Sobolev IPM compares the mean discrepancy of two distributions for functions (critic) restricted to a Sobolev ball defined with respect to a dominant measure $μ$. We show that the Sobolev IPM compares two distributions in high dimensions based on weighted conditional Cumulative Distribution Functions (CDF) of each coordinate on a leave one out basis. The Dominant measure $μ$ plays a crucial role as it defines the support on which conditional CDFs are compared. Sobolev IPM can be seen as an extension of the one dimensional Von-Mises Cramér statistics to high dimensional distributions. We show how Sobolev IPM can be used to train Generative Adversarial Networks (GANs). We then exploit the intrinsic conditioning implied by Sobolev IPM in text generation. Finally we show that a variant of Sobolev GAN achieves competitive results in semi-supervised learning on CIFAR-10, thanks to the smoothness enforced on the critic by Sobolev GAN which relates to Laplacian regularization.