Incremental Gauss--Newton Methods with Superlinear Convergence Rates
This work addresses a problem in optimization for researchers and practitioners dealing with large-scale nonlinear equations, offering incremental improvements over existing methods.
The paper tackles the challenge of solving large-scale nonlinear equations with Hölder continuous Jacobians by introducing an Incremental Gauss–Newton method that achieves explicit superlinear convergence rates, outperforming existing methods with only linear rates, and includes a mini-batch extension for even faster convergence.
This paper addresses the challenge of solving large-scale nonlinear equations with Hölder continuous Jacobians. We introduce a novel Incremental Gauss--Newton (IGN) method within explicit superlinear convergence rate, which outperforms existing methods that only achieve linear convergence rate. In particular, we formulate our problem by the nonlinear least squares with finite-sum structure, and our method incrementally iterates with the information of one component in each round. We also provide a mini-batch extension to our IGN method that obtains an even faster superlinear convergence rate. Furthermore, we conduct numerical experiments to show the advantages of the proposed methods.