Fast-Slow Co-advancing Optimizer: Toward Harmonious Adversarial Training of GAN
This addresses training instability in GANs for researchers and practitioners, but it is incremental as it builds on existing optimization methods.
The paper tackles the sensitivity of GAN training to data properties and hyperparameters, which causes oscillations and convergence issues, by developing the Fast-Slow Co-advancing Optimizer (FSCO) that uses reinforcement learning to control step size, resulting in improved stability and reduced sensitivity.
Up to now, the training processes of typical Generative Adversarial Networks (GANs) are still particularly sensitive to data properties and hyperparameters, which may lead to severe oscillations, difficulties in convergence, or even failures to converge, especially when the overall variances of the training sets are large. These phenomena are often attributed to the training characteristics of such networks. Aiming at the problem, this paper develops a new intelligent optimizer, Fast-Slow Co-advancing Optimizer (FSCO), which employs reinforcement learning in the training process of GANs to make training easier. Specifically, this paper allows the training step size to be controlled by an agent to improve training stability, and makes the training process more intelligent with variable learning rates, making GANs less sensitive to step size. Experiments have been conducted on three benchmark datasets to verify the effectiveness of the developed FSCO.