Accelerated Stochastic Gradient Descent for Minimizing Finite Sums
This work addresses optimization efficiency for machine learning practitioners dealing with large-scale convex problems, representing an incremental improvement over existing methods.
The paper tackles the problem of minimizing finite sums of smooth convex functions by proposing a method that combines accelerated gradient descent with stochastic variance reduction in a mini-batch setting, achieving lower overall complexity for non-strongly convex problems and fast convergence for strongly convex ones.
We propose an optimization method for minimizing the finite sums of smooth convex functions. Our method incorporates an accelerated gradient descent (AGD) and a stochastic variance reduction gradient (SVRG) in a mini-batch setting. Unlike SVRG, our method can be directly applied to non-strongly and strongly convex problems. We show that our method achieves a lower overall complexity than the recently proposed methods that supports non-strongly convex problems. Moreover, this method has a fast rate of convergence for strongly convex problems. Our experiments show the effectiveness of our method.