On the Suboptimality of Negative Momentum for Minimax Optimization
This work addresses algorithm design challenges in game optimization for machine learning, offering incremental theoretical insights into negative momentum's performance.
The paper tackles the convergence rate of negative momentum in smooth and strongly-convex strongly-concave minimax games, showing that it accelerates convergence locally but with a suboptimal rate, providing the first explicit convergence rate in this setting.
Smooth game optimization has recently attracted great interest in machine learning as it generalizes the single-objective optimization paradigm. However, game dynamics is more complex due to the interaction between different players and is therefore fundamentally different from minimization, posing new challenges for algorithm design. Notably, it has been shown that negative momentum is preferred due to its ability to reduce oscillation in game dynamics. Nevertheless, the convergence rate of negative momentum was only established in simple bilinear games. In this paper, we extend the analysis to smooth and strongly-convex strongly-concave minimax games by taking the variational inequality formulation. By connecting momentum method with Chebyshev polynomials, we show that negative momentum accelerates convergence of game dynamics locally, though with a suboptimal rate. To the best of our knowledge, this is the \emph{first work} that provides an explicit convergence rate for negative momentum in this setting.