Multilevel Objective-Function-Free Optimization with an Application to Neural Networks Training
This work addresses optimization challenges in noisy environments for machine learning practitioners, presenting an incremental improvement over existing methods like AdaGrad.
The authors tackled the problem of unconstrained nonlinear optimization without evaluating the objective function, aiming to reduce sensitivity to noise and computational cost, and demonstrated its application in training deep neural networks with improved noise robustness.
A class of multi-level algorithms for unconstrained nonlinear optimization is presented which does not require the evaluation of the objective function. The class contains the momentum-less AdaGrad method as a particular (single-level) instance. The choice of avoiding the evaluation of the objective function is intended to make the algorithms of the class less sensitive to noise, while the multi-level feature aims at reducing their computational cost. The evaluation complexity of these algorithms is analyzed and their behaviour in the presence of noise is then illustrated in the context of training deep neural networks for supervised learning applications.