Improved Optimization of Finite Sums with Minibatch Stochastic Variance Reduced Proximal Iterations
This work addresses optimization efficiency for machine learning practitioners, but it appears incremental as it builds on existing variance reduction techniques.
The paper tackles empirical risk minimization by introducing minibatch stochastic optimization methods that use variance reduction and higher-order information to accelerate convergence, achieving improved iteration complexity for quadratic objectives and demonstrating empirical advantages over existing methods.
We present novel minibatch stochastic optimization methods for empirical risk minimization problems, the methods efficiently leverage variance reduced first-order and sub-sampled higher-order information to accelerate the convergence speed. For quadratic objectives, we prove improved iteration complexity over state-of-the-art under reasonable assumptions. We also provide empirical evidence of the advantages of our method compared to existing approaches in the literature.