Online Kernel based Generative Adversarial Networks
This addresses training challenges in GANs for researchers and practitioners, offering a more stable and theoretically grounded approach, though it is incremental as it builds on existing GAN frameworks.
The paper tackles training instability in Generative Adversarial Networks (GANs), such as mode collapse and cycling, by proposing OKGAN, which uses a kernel-based non-parametric discriminator. It shows that OKGAN mitigates these issues, performs dramatically better on synthetic data with respect to reverse KL-divergence, and achieves comparable performance on vision datasets like MNIST, SVHN, and CelebA.
One of the major breakthroughs in deep learning over the past five years has been the Generative Adversarial Network (GAN), a neural network-based generative model which aims to mimic some underlying distribution given a dataset of samples. In contrast to many supervised problems, where one tries to minimize a simple objective function of the parameters, GAN training is formulated as a min-max problem over a pair of network parameters. While empirically GANs have shown impressive success in several domains, researchers have been puzzled by unusual training behavior, including cycling so-called mode collapse. In this paper, we begin by providing a quantitative method to explore some of the challenges in GAN training, and we show empirically how this relates fundamentally to the parametric nature of the discriminator network. We propose a novel approach that resolves many of these issues by relying on a kernel-based non-parametric discriminator that is highly amenable to online training---we call this the Online Kernel-based Generative Adversarial Networks (OKGAN). We show empirically that OKGANs mitigate a number of training issues, including mode collapse and cycling, and are much more amenable to theoretical guarantees. OKGANs empirically perform dramatically better, with respect to reverse KL-divergence, than other GAN formulations on synthetic data; on classical vision datasets such as MNIST, SVHN, and CelebA, show comparable performance.