Generalization of GANs and overparameterized models under Lipschitz continuity
This provides a theoretical foundation for understanding generalization in overparameterized models like GANs and deep neural networks, addressing a long-standing open problem in machine learning.
The paper tackles the lack of theoretical explanation for the success of GANs and deep neural networks by introducing a Lipschitz theory to analyze generalization, showing that penalizing the Lipschitz constant improves GAN generalization and that deep models can generalize well without the curse of dimensionality when using techniques like Dropout or spectral normalization.
Generative adversarial networks (GANs) are so complex that the existing learning theories do not provide a satisfactory explanation for why GANs have great success in practice. The same situation also remains largely open for deep neural networks. To fill this gap, we introduce a Lipschitz theory to analyze generalization. We demonstrate its simplicity by analyzing generalization and consistency of overparameterized neural networks. We then use this theory to derive Lipschitz-based generalization bounds for GANs. Our bounds show that penalizing the Lipschitz constant of the GAN loss can improve generalization. This result answers the long mystery of why the popular use of Lipschitz constraint for GANs often leads to great success, empirically without a solid theory. Finally but surprisingly, we show that, when using Dropout or spectral normalization, both \emph{truly deep} neural networks and GANs can generalize well without the curse of dimensionality.