A fast quasi-Newton-type method for large-scale stochastic optimisation
This work addresses the need for efficient optimization algorithms in machine learning and data science, but it is incremental as it builds on existing quasi-Newton methods with slight modifications.
The authors tackled the problem of large-scale stochastic optimization by proposing a fast quasi-Newton-type method that computes search directions using a direct least-squares approach with a low-dimensional Cholesky factor, combined with a stochastic line search. The result showed improved performance on real-world benchmark problems compared to established methods, though no concrete numbers were provided.
During recent years there has been an increased interest in stochastic adaptations of limited memory quasi-Newton methods, which compared to pure gradient-based routines can improve the convergence by incorporating second order information. In this work we propose a direct least-squares approach conceptually similar to the limited memory quasi-Newton methods, but that computes the search direction in a slightly different way. This is achieved in a fast and numerically robust manner by maintaining a Cholesky factor of low dimension. This is combined with a stochastic line search relying upon fulfilment of the Wolfe condition in a backtracking manner, where the step length is adaptively modified with respect to the optimisation progress. We support our new algorithm by providing several theoretical results guaranteeing its performance. The performance is demonstrated on real-world benchmark problems which shows improved results in comparison with already established methods.