EA-CG: An Approximate Second-Order Method for Training Fully-Connected Neural Networks
This work addresses the computational bottlenecks in training neural networks, but it is incremental as it builds on existing second-order optimization techniques.
The paper tackled the problem of training fully-connected neural networks by proposing an approximate second-order method with a memory-efficient Hessian approximation and a conjugate gradient-based approach, resulting in significant reductions in space and time complexity as shown in empirical studies.
For training fully-connected neural networks (FCNNs), we propose a practical approximate second-order method including: 1) an approximation of the Hessian matrix and 2) a conjugate gradient (CG) based method. Our proposed approximate Hessian matrix is memory-efficient and can be applied to any FCNNs where the activation and criterion functions are twice differentiable. We devise a CG-based method incorporating one-rank approximation to derive Newton directions for training FCNNs, which significantly reduces both space and time complexity. This CG-based method can be employed to solve any linear equation where the coefficient matrix is Kronecker-factored, symmetric and positive definite. Empirical studies show the efficacy and efficiency of our proposed method.