Explicit Second-Order Min-Max Optimization: Practical Algorithms and Complexity Analysis
For researchers in optimization and machine learning, this work advances second-order methods for min-max problems by providing practical algorithms with improved complexity guarantees and empirical performance.
This paper develops inexact regularized Newton-type methods for convex-concave min-max optimization, achieving an O(ε^{-2/3}) iteration complexity to find an ε-saddle point, improving over existing second-order methods by reducing the number of Schur decompositions by a factor of O(log log(1/ε)). Experiments on synthetic benchmarks and AUC maximization tasks show practical efficiency gains over first-order methods.
We propose and analyze several inexact regularized Newton-type methods for finding a global saddle point of convex-concave unconstrained min-max optimization problems. Compared to first-order methods, our understanding of second-order methods for min-max optimization is relatively limited, as obtaining global rates of convergence with second-order information can be much more involved. In this paper, we examine how second-order information is used to speed up extra-gradient methods, even under inexactness. In particular, we show that the proposed methods generate iterates that remain within a bounded set and that the averaged iterates converge to an $ε$-saddle point within $O(ε^{-2/3})$ iterations in terms of a restricted gap function. We also provide a simple routine for solving the subproblem at each iteration, requiring a single Schur decomposition and $O(\log\log(1/ε))$ calls to a linear system solver in a quasi-upper-triangular system. Thus, our method improves the existing line-search-based second-order min-max optimization methods by shaving off an $O(\log\log(1/ε))$ factor in the required number of Schur decompositions. Finally, we evaluate our method on both synthetic benchmarks and a real-world application arising from AUC maximization on standard LIBSVM datasets, and find that the proposed second-order approach delivers stronger practical efficiency than representative first-order methods on these problems.