A Momentum Accelerated Adaptive Cubic Regularization Method for Nonconvex Optimization
This work addresses convergence speed for researchers and practitioners in optimization, though it is incremental as it builds on existing cubic regularization methods.
The paper tackled the problem of slow convergence in non-convex optimization by developing a momentum-accelerated adaptive cubic regularization method (ARCm), which reduced the number of iterations by 10% to 50% compared to the most competitive existing method in experiments.
The cubic regularization method (CR) and its adaptive version (ARC) are popular Newton-type methods in solving unconstrained non-convex optimization problems, due to its global convergence to local minima under mild conditions. The main aim of this paper is to develop a momentum-accelerated adaptive cubic regularization method (ARCm) to improve the convergent performance. With the proper choice of momentum step size, we show the global convergence of ARCm and the local convergence can also be guaranteed under the \KL property. Such global and local convergence can also be established when inexact solvers with low computational costs are employed in the iteration procedure. Numerical results for non-convex logistic regression and robust linear regression models are reported to demonstrate that the proposed ARCm significantly outperforms state-of-the-art cubic regularization methods (e.g., CR, momentum-based CR, ARC) and the trust region method. In particular, the number of iterations required by ARCm is less than 10\% to 50\% required by the most competitive method (ARC) in the experiments.