Double Descent and Other Interpolation Phenomena in GANs
This addresses the problem of improving training efficiency and generalization in GANs for machine learning practitioners, though it is incremental as it builds on existing overparameterization and interpolation concepts.
The paper investigates overparameterization in GANs, showing it can enhance generalization and speed up training, and introduces a pseudo-supervised learning method that exhibits double or triple descent in generalization errors, with this approach accelerating training while matching or surpassing performance without pseudo-supervision.
We study overparameterization in generative adversarial networks (GANs) that can interpolate the training data. We show that overparameterization can improve generalization performance and accelerate the training process. We study the generalization error as a function of latent space dimension and identify two main behaviors, depending on the learning setting. First, we show that overparameterized generative models that learn distributions by minimizing a metric or $f$-divergence do not exhibit double descent in generalization errors; specifically, all the interpolating solutions achieve the same generalization error. Second, we develop a novel pseudo-supervised learning approach for GANs where the training utilizes pairs of fabricated (noise) inputs in conjunction with real output samples. Our pseudo-supervised setting exhibits double descent (and in some cases, triple descent) of generalization errors. We combine pseudo-supervision with overparameterization (i.e., overly large latent space dimension) to accelerate training while matching or even surpassing generalization performance without pseudo-supervision. While our analysis focuses mostly on linear models, we also apply important insights for improving generalization of nonlinear, multilayer GANs.