Improving Levenberg-Marquardt Algorithm for Neural Networks
This work addresses optimization efficiency for neural network training, but appears incremental as it builds on existing LM methods with enhancements.
The paper tackled improving the Levenberg-Marquardt algorithm for neural networks by incorporating adaptive momentum, learning rate line search, and uphill step acceptance, resulting in performance comparisons with first-order and second-order algorithms like SGD, Adam, L-BFGS, Hessian-Free, and KFAC.
We explore the usage of the Levenberg-Marquardt (LM) algorithm for regression (non-linear least squares) and classification (generalized Gauss-Newton methods) tasks in neural networks. We compare the performance of the LM method with other popular first-order algorithms such as SGD and Adam, as well as other second-order algorithms such as L-BFGS , Hessian-Free and KFAC. We further speed up the LM method by using adaptive momentum, learning rate line search, and uphill step acceptance.