Jun-Lin Wang

Zi Xu, Zi-Qi Wang, Jun-Lin Wang et al.

In this paper, we consider a class of nonconvex-nonconcave minimax problems, i.e., NC-PL minimax problems, whose objective functions satisfy the Polyak-Łojasiewicz (PL) condition with respect to the inner variable. We propose a zeroth-order alternating gradient descent ascent (ZO-AGDA) algorithm and a zeroth-order variance reduced alternating gradient descent ascent (ZO-VRAGDA) algorithm for solving NC-PL minimax problem under the deterministic and the stochastic setting, respectively. The total number of function value queries to obtain an $ε$-stationary point of ZO-AGDA and ZO-VRAGDA algorithm for solving NC-PL minimax problem is upper bounded by $\mathcal{O}(\varepsilon^{-2})$ and $\mathcal{O}(\varepsilon^{-3})$, respectively. To the best of our knowledge, they are the first two zeroth-order algorithms with the iteration complexity gurantee for solving NC-PL minimax problems.

Jun-Lin Wang, Zi Xu

In this paper, we study second-order algorithms for solving nonconvex-strongly concave minimax problems, which have attracted much attention in recent years in many fields, especially in machine learning.We propose a gradient norm regularized trust-region (GRTR) algorithm to solve nonconvex-strongly concave minimax problems, where the objective function of the trust-region subproblem in each iteration uses a regularized version of the Hessian matrix, and the regularization coefficient and the radius of the ball constraint are proportional to the square root of the gradient norm. The iteration complexity of the proposed GRTR algorithm to obtain an $O(ε,\sqrtε)$-second-order stationary point is proved to be upper bounded by $\tilde{O}(\ell^{1.5}ρ^{0.5}μ^{-1.5}ε^{-1.5})$, where $μ$ is the strong concave coefficient, $\ell$ and $ρ$ are the Lipschitz constant of the gradient and Jacobian matrix respectively, which matches the best known iteration complexity of second-order methods for solving nonconvex-strongly concave minimax problems. We further propose a Levenberg-Marquardt algorithm with a gradient norm regularization coefficient and use the negative curvature direction to correct the iteration direction (LMNegCur), which does not need to solve the trust-region subproblem at each iteration. We also prove that the LMNegCur algorithm achieves an $O(ε,\sqrtε)$-second-order stationary point within $\tilde{O}(\ell^{1.5}ρ^{0.5}μ^{-1.5}ε^{-1.5})$ number of iterations.The inexact variants of both algorithms can still obtain $O(ε,\sqrtε)$-second-order stationary points with high probability, but only require $\tilde{O}(\ell^{2.25}ρ^{0.25}μ^{-1.75}ε^{-1.75})$ Hessian-vector products and $\tilde{O}(\ell^{2}ρ^{0.5}μ^{-2}ε^{-1.5})$ gradient ascent steps.