ODE Analysis of Stochastic Gradient Methods with Optimism and Anchoring for Minimax Problems
This work addresses the theoretical gap in training dynamics for minimax problems like GANs, though it is incremental as it builds on existing methods with new analyses and a minor variant.
The paper tackled the problem of understanding and ensuring convergence in minimax optimization, particularly for GAN training, by analyzing variants of simultaneous gradient descent and showing that they achieve last-iterate convergence under convex-concave assumptions.
Despite remarkable empirical success, the training dynamics of generative adversarial networks (GAN), which involves solving a minimax game using stochastic gradients, is still poorly understood. In this work, we analyze last-iterate convergence of simultaneous gradient descent (simGD) and its variants under the assumption of convex-concavity, guided by a continuous-time analysis with differential equations. First, we show that simGD, as is, converges with stochastic sub-gradients under strict convexity in the primal variable. Second, we generalize optimistic simGD to accommodate an optimism rate separate from the learning rate and show its convergence with full gradients. Finally, we present anchored simGD, a new method, and show convergence with stochastic subgradients.