Negative Momentum for Improved Game Dynamics
This addresses training instability in generative adversarial networks (GANs) and other game-based ML systems, offering a practical improvement.
The paper tackles the problem of unstable training dynamics in differentiable games like GANs by analyzing gradient-based methods with momentum, proving that alternating updates are more stable than simultaneous ones and showing that alternating gradient updates with negative momentum achieve convergence in difficult adversarial problems, including saturating GANs.
Games generalize the single-objective optimization paradigm by introducing different objective functions for different players. Differentiable games often proceed by simultaneous or alternating gradient updates. In machine learning, games are gaining new importance through formulations like generative adversarial networks (GANs) and actor-critic systems. However, compared to single-objective optimization, game dynamics are more complex and less understood. In this paper, we analyze gradient-based methods with momentum on simple games. We prove that alternating updates are more stable than simultaneous updates. Next, we show both theoretically and empirically that alternating gradient updates with a negative momentum term achieves convergence in a difficult toy adversarial problem, but also on the notoriously difficult to train saturating GANs.