A Solvable High-Dimensional Model of GAN
This provides a foundational theoretical framework for understanding GAN training dynamics, though it is incremental as it focuses on a simple model.
The authors tackled the theoretical analysis of training dynamics in a single-layer GAN with high-dimensional input, proving that macroscopic quantities converge to a deterministic ODE while microscopic states remain stochastic, described by an SDE, and showing that noise level critically affects convergence, with too strong noise preventing feature recovery and too weak causing oscillation.
We present a theoretical analysis of the training process for a single-layer GAN fed by high-dimensional input data. The training dynamics of the proposed model at both microscopic and macroscopic scales can be exactly analyzed in the high-dimensional limit. In particular, we prove that the macroscopic quantities measuring the quality of the training process converge to a deterministic process characterized by an ordinary differential equation (ODE), whereas the microscopic states containing all the detailed weights remain stochastic, whose dynamics can be described by a stochastic differential equation (SDE). This analysis provides a new perspective different from recent analyses in the limit of small learning rate, where the microscopic state is always considered deterministic, and the contribution of noise is ignored. From our analysis, we show that the level of the background noise is essential to the convergence of the training process: setting the noise level too strong leads to failure of feature recovery, whereas setting the noise too weak causes oscillation. Although this work focuses on a simple copy model of GAN, we believe the analysis methods and insights developed here would prove useful in the theoretical understanding of other variants of GANs with more advanced training algorithms.