FINDER: Stochastic Mirroring of Noisy Quasi-Newton Search and Deep Network Training
This work addresses large-scale optimization challenges in machine learning, such as training deep networks, but appears incremental as it builds on existing quasi-Newton and stochastic filtering ideas.
The authors tackled the problem of optimizing non-convex and non-smooth functions in high-dimensional spaces by developing FINDER, a stochastic optimizer that combines noise-assisted global search with Newton-like local convergence, achieving promising performance compared to Adam and other methods on benchmark functions and deep network tasks.
Our proposal is on a new stochastic optimizer for non-convex and possibly non-smooth objective functions typically defined over large dimensional design spaces. Towards this, we have tried to bridge noise-assisted global search and faster local convergence, the latter being the characteristic feature of a Newton-like search. Our specific scheme -- acronymed FINDER (Filtering Informed Newton-like and Derivative-free Evolutionary Recursion), exploits the nonlinear stochastic filtering equations to arrive at a derivative-free update that has resemblance with the Newton search employing the inverse Hessian of the objective function. Following certain simplifications of the update to enable a linear scaling with dimension and a few other enhancements, we apply FINDER to a range of problems, starting with some IEEE benchmark objective functions to a couple of archetypal data-driven problems in deep networks to certain cases of physics-informed deep networks. The performance of the new method vis-á-vis the well-known Adam and a few others bears evidence to its promise and potentialities for large dimensional optimization problems of practical interest.