The Unusual Effectiveness of Averaging in GAN Training
This work addresses instability in GAN training for generative modeling, providing incremental improvements with theoretical and empirical support.
The paper tackled the problem of stabilizing GAN training by analyzing parameter averaging techniques, showing that both Moving Average (MA) and Exponential Moving Average (EMA) improve inception and FID scores across various datasets, achieving state-of-the-art results on CIFAR-10 and producing clean CelebA images.
We examine two different techniques for parameter averaging in GAN training. Moving Average (MA) computes the time-average of parameters, whereas Exponential Moving Average (EMA) computes an exponentially discounted sum. Whilst MA is known to lead to convergence in bilinear settings, we provide the -- to our knowledge -- first theoretical arguments in support of EMA. We show that EMA converges to limit cycles around the equilibrium with vanishing amplitude as the discount parameter approaches one for simple bilinear games and also enhances the stability of general GAN training. We establish experimentally that both techniques are strikingly effective in the non-convex-concave GAN setting as well. Both improve inception and FID scores on different architectures and for different GAN objectives. We provide comprehensive experimental results across a range of datasets -- mixture of Gaussians, CIFAR-10, STL-10, CelebA and ImageNet -- to demonstrate its effectiveness. We achieve state-of-the-art results on CIFAR-10 and produce clean CelebA face images.\footnote{~The code is available at \url{https://github.com/yasinyazici/EMA_GAN}}