Near-Optimal Algorithms for Making the Gradient Small in Stochastic Minimax Optimization
This work addresses the inefficiency of extending deterministic methods to stochastic settings in minimax optimization, offering near-optimal algorithms for researchers and practitioners in machine learning and optimization.
The paper tackles the problem of finding near-stationary points in stochastic minimax optimization by proposing the Recursive Anchored IteratioN (RAIN) algorithm, which achieves near-optimal stochastic first-order oracle (SFO) complexity for convex-concave, strongly-convex-strongly-concave, and structured nonconvex-nonconcave cases.
We study the problem of finding a near-stationary point for smooth minimax optimization. The recently proposed extra anchored gradient (EAG) methods achieve the optimal convergence rate for the convex-concave minimax problem in the deterministic setting. However, the direct extension of EAG to stochastic optimization is not efficient. In this paper, we design a novel stochastic algorithm called Recursive Anchored IteratioN (RAIN). We show that the RAIN achieves near-optimal stochastic first-order oracle (SFO) complexity for stochastic minimax optimization in both convex-concave and strongly-convex-strongly-concave cases. In addition, we extend the idea of RAIN to solve structured nonconvex-nonconcave minimax problem and it also achieves near-optimal SFO complexity.