Newton-CG methods for nonconvex unconstrained optimization with Hölder continuous Hessian
This addresses optimization efficiency for nonconvex problems in machine learning and scientific computing, offering a parameter-free method that is incremental but improves practical performance.
The paper tackles nonconvex unconstrained optimization with Hölder continuous Hessian by proposing a Newton-CG method that finds approximate first- and second-order stationary points, achieving the best-known iteration and operation complexity as the first parameter-free second-order method, with preliminary numerical results showing superior performance over a regularized Newton method.
In this paper we consider a nonconvex unconstrained optimization problem minimizing a twice differentiable objective function with Hölder continuous Hessian. Specifically, we first propose a Newton-conjugate gradient (Newton-CG) method for finding an approximate first- and second-order stationary point of this problem, assuming the associated the Hölder parameters are explicitly known. Then we develop a parameter-free Newton-CG method without requiring any prior knowledge of these parameters. To the best of our knowledge, this method is the first parameter-free second-order method achieving the best-known iteration and operation complexity for finding an approximate first- and second-order stationary point of this problem. Finally, we present preliminary numerical results to demonstrate the superior practical performance of our parameter-free Newton-CG method over a well-known regularized Newton method.