A modified limited memory Nesterov's accelerated quasi-Newton
This is an incremental improvement for optimizing neural networks by addressing a specific computational bottleneck.
The paper tackled the computational cost of Nesterov's accelerated quasi-Newton methods by extending a momentum-based approximation to limited memory variants, resulting in reduced gradient calculations per iteration.
The Nesterov's accelerated quasi-Newton (L)NAQ method has shown to accelerate the conventional (L)BFGS quasi-Newton method using the Nesterov's accelerated gradient in several neural network (NN) applications. However, the calculation of two gradients per iteration increases the computational cost. The Momentum accelerated Quasi-Newton (MoQ) method showed that the Nesterov's accelerated gradient can be approximated as a linear combination of past gradients. This abstract extends the MoQ approximation to limited memory NAQ and evaluates the performance on a function approximation problem.