Inexact SARAH Algorithm for Stochastic Optimization
This work addresses a bottleneck in stochastic optimization for machine learning practitioners by extending applicability beyond finite sum problems, though it is incremental as it builds on existing SARAH and SVRG methods.
The paper tackles the problem of stochastic optimization by developing an inexact variant of the SARAH algorithm (iSARAH) that eliminates the need for exact gradient computations, allowing application to general expectation minimization problems. It achieves the best known complexity among stochastic methods for non-strongly convex functions under reasonable assumptions.
We develop and analyze a variant of the SARAH algorithm, which does not require computation of the exact gradient. Thus this new method can be applied to general expectation minimization problems rather than only finite sum problems. While the original SARAH algorithm, as well as its predecessor, SVRG, require an exact gradient computation on each outer iteration, the inexact variant of SARAH (iSARAH), which we develop here, requires only stochastic gradient computed on a mini-batch of sufficient size. The proposed method combines variance reduction via sample size selection and iterative stochastic gradient updates. We analyze the convergence rate of the algorithms for strongly convex and non-strongly convex cases, under smooth assumption with appropriate mini-batch size selected for each case. We show that with an additional, reasonable, assumption iSARAH achieves the best known complexity among stochastic methods in the case of non-strongly convex stochastic functions.