Convergence Analysis of Block Newton Methods for 1D Shallow Neural Network Approximation
This work addresses the optimization of neural network parameters for approximation problems, but it is incremental as it builds on existing block Newton methods.
This paper analyzes the local convergence of block Newton methods for approximating functions and diffusion-reaction problems using one-dimensional shallow neural networks, establishing convergence under reasonable assumptions and showing that the reduced block Newton method can reduce parameters during optimization.
This paper analyzes local convergence of the block Newton (BN) method introduced in [5, 6] for one-dimensional shallow neural network approximation to functions and diffusion-reaction problems. The BN method consists of the 2x2 block nonlinear Gauss-Seidel, linear Gauss-Seidel, or Jacobi method for outer iteration and the Newton method for inner iteration. The blocks are corresponding to the linear and the nonlinear parameters. Under some reasonable assumptions, we establish local convergence of the BN methods as well as the reduced BN (rBN) method for one-dimensional diffusion-reaction problems and least-squares function approximation. Unlike common optimization methods, the rBN allows for the reduction of the number of parameters during the optimization process when some neurons contribute little to the approximation or are at nearly optimal locations.