Data Interpolants -- That's What Discriminators in Higher-order Gradient-regularized GANs Are
This work addresses a specific bottleneck in GAN training by providing a closed-form solution for discriminator optimization, which is incremental but offers practical improvements for generative modeling tasks.
The paper tackles the problem of optimizing the discriminator in GANs with higher-order gradient regularization, showing analytically that it reduces to interpolation in n-dimensions, and demonstrates through experiments on multivariate Gaussians that implementing the optimal discriminator via polyharmonic RBF interpolation yields superior performance compared to arbitrary architectures.
We consider the problem of optimizing the discriminator in generative adversarial networks (GANs) subject to higher-order gradient regularization. We show analytically, via the least-squares (LSGAN) and Wasserstein (WGAN) GAN variants, that the discriminator optimization problem is one of interpolation in $n$-dimensions. The optimal discriminator, derived using variational Calculus, turns out to be the solution to a partial differential equation involving the iterated Laplacian or the polyharmonic operator. The solution is implementable in closed-form via polyharmonic radial basis function (RBF) interpolation. In view of the polyharmonic connection, we refer to the corresponding GANs as Poly-LSGAN and Poly-WGAN. Through experimental validation on multivariate Gaussians, we show that implementing the optimal RBF discriminator in closed-form, with penalty orders $m \approx\lceil \frac{n}{2} \rceil $, results in superior performance, compared to training GAN with arbitrarily chosen discriminator architectures. We employ the Poly-WGAN discriminator to model the latent space distribution of the data with encoder-decoder-based GAN flavors such as Wasserstein autoencoders.