Quasi-Newton Methods: A New Direction
This work provides a new theoretical foundation for optimization algorithms, potentially improving efficiency in numerical optimization tasks.
The paper tackles the interpretation of quasi-Newton methods as Bayesian linear regression approximations, revealing shortcomings in classical algorithms and leading to a novel nonparametric method that uses information more efficiently with similar computational cost.
Four decades after their invention, quasi-Newton methods are still state of the art in unconstrained numerical optimization. Although not usually interpreted thus, these are learning algorithms that fit a local quadratic approximation to the objective function. We show that many, including the most popular, quasi-Newton methods can be interpreted as approximations of Bayesian linear regression under varying prior assumptions. This new notion elucidates some shortcomings of classical algorithms, and lights the way to a novel nonparametric quasi-Newton method, which is able to make more efficient use of available information at computational cost similar to its predecessors.