TiAda: A Time-scale Adaptive Algorithm for Nonconvex Minimax Optimization
This work addresses a bottleneck in machine learning applications like GANs by providing a parameter-agnostic solution for nonconvex minimax optimization, though it is incremental as it builds on existing gradient descent ascent methods.
The authors tackled the challenge of hyperparameter tuning in nonconvex minimax optimization by proposing TiAda, a time-scale adaptive algorithm that automatically adjusts to time-scale separation, achieving near-optimal complexities in deterministic and stochastic settings.
Adaptive gradient methods have shown their ability to adjust the stepsizes on the fly in a parameter-agnostic manner, and empirically achieve faster convergence for solving minimization problems. When it comes to nonconvex minimax optimization, however, current convergence analyses of gradient descent ascent (GDA) combined with adaptive stepsizes require careful tuning of hyper-parameters and the knowledge of problem-dependent parameters. Such a discrepancy arises from the primal-dual nature of minimax problems and the necessity of delicate time-scale separation between the primal and dual updates in attaining convergence. In this work, we propose a single-loop adaptive GDA algorithm called TiAda for nonconvex minimax optimization that automatically adapts to the time-scale separation. Our algorithm is fully parameter-agnostic and can achieve near-optimal complexities simultaneously in deterministic and stochastic settings of nonconvex-strongly-concave minimax problems. The effectiveness of the proposed method is further justified numerically for a number of machine learning applications.