Demystifying MMD GANs
This work addresses theoretical and practical challenges in GAN training for researchers and practitioners, offering incremental improvements in understanding and efficiency.
The paper clarifies theoretical issues regarding bias in GAN loss functions for MMD and Wasserstein GANs, showing that gradient estimators are unbiased but learning a discriminator from samples leads to biased gradients for the generator. It also introduces the Kernel Inception Distance as an improved convergence measure and demonstrates that MMD GANs achieve matching performance with simpler, faster training using smaller critic networks.
We investigate the training and performance of generative adversarial networks using the Maximum Mean Discrepancy (MMD) as critic, termed MMD GANs. As our main theoretical contribution, we clarify the situation with bias in GAN loss functions raised by recent work: we show that gradient estimators used in the optimization process for both MMD GANs and Wasserstein GANs are unbiased, but learning a discriminator based on samples leads to biased gradients for the generator parameters. We also discuss the issue of kernel choice for the MMD critic, and characterize the kernel corresponding to the energy distance used for the Cramer GAN critic. Being an integral probability metric, the MMD benefits from training strategies recently developed for Wasserstein GANs. In experiments, the MMD GAN is able to employ a smaller critic network than the Wasserstein GAN, resulting in a simpler and faster-training algorithm with matching performance. We also propose an improved measure of GAN convergence, the Kernel Inception Distance, and show how to use it to dynamically adapt learning rates during GAN training.