A Simple Practical Accelerated Method for Finite Sums
This work addresses optimization efficiency for machine learning practitioners dealing with finite-sum problems, offering a simpler and more versatile accelerated method.
The paper tackles the problem of optimizing finite sums (like empirical risk minimization) by introducing a novel method that achieves accelerated convergence rates for strongly convex smooth problems with only one parameter (step size), making it simpler than existing accelerated methods. The method can also handle non-smooth terms, expanding its applicability to areas traditionally using operator splitting methods.
We describe a novel optimization method for finite sums (such as empirical risk minimization problems) building on the recently introduced SAGA method. Our method achieves an accelerated convergence rate on strongly convex smooth problems. Our method has only one parameter (a step size), and is radically simpler than other accelerated methods for finite sums. Additionally it can be applied when the terms are non-smooth, yielding a method applicable in many areas where operator splitting methods would traditionally be applied.