Fast Incremental Method for Nonconvex Optimization
This work addresses optimization challenges in machine learning for researchers and practitioners, but it is incremental as it extends existing methods to nonconvex settings.
The paper tackles the problem of optimizing nonconvex functions using incremental aggregated gradient methods, specifically analyzing the SAGA algorithm and showing it converges faster than gradient descent and stochastic gradient descent to stationary points, with linear convergence to global optima for a special class of problems.
We analyze a fast incremental aggregated gradient method for optimizing nonconvex problems of the form $\min_x \sum_i f_i(x)$. Specifically, we analyze the SAGA algorithm within an Incremental First-order Oracle framework, and show that it converges to a stationary point provably faster than both gradient descent and stochastic gradient descent. We also discuss a Polyak's special class of nonconvex problems for which SAGA converges at a linear rate to the global optimum. Finally, we analyze the practically valuable regularized and minibatch variants of SAGA. To our knowledge, this paper presents the first analysis of fast convergence for an incremental aggregated gradient method for nonconvex problems.