Accelerated Stochastic Min-Max Optimization Based on Bias-corrected Momentum
This work addresses a theoretical bottleneck in optimization for machine learning, offering incremental improvements in convergence rates under specific conditions.
The paper tackles the problem of achieving faster convergence rates in stochastic min-max optimization for nonconvex-strongly-concave or nonconvex-PL settings, introducing bias-corrected momentum algorithms that reduce iteration complexity to O(ε^{-3}) and validating them on robust logistic regression with real-world datasets.
Lower-bound analyses for nonconvex strongly-concave minimax optimization problems have shown that stochastic first-order algorithms require at least $\mathcal{O}(\varepsilon^{-4})$ oracle complexity to find an $\varepsilon$-stationary point. Some works indicate that this complexity can be improved to $\mathcal{O}(\varepsilon^{-3})$ when the loss gradient is Lipschitz continuous. The question of achieving enhanced convergence rates under distinct conditions, remains unresolved. In this work, we address this question for optimization problems that are nonconvex in the minimization variable and strongly concave or Polyak-Lojasiewicz (PL) in the maximization variable. We introduce novel bias-corrected momentum algorithms utilizing efficient Hessian-vector products. We establish convergence conditions and demonstrate a lower iteration complexity of $\mathcal{O}(\varepsilon^{-3})$ for the proposed algorithms. The effectiveness of the method is validated through applications to robust logistic regression using real-world datasets.