A variable metric mini-batch proximal stochastic recursive gradient algorithm with diagonal Barzilai-Borwein stepsize
This work addresses optimization efficiency for machine learning practitioners, but it is incremental as it builds on existing variable metric and stochastic gradient methods.
The authors tackled the problem of composite optimization by proposing a variable metric mini-batch proximal stochastic recursive gradient algorithm with a diagonal Barzilai-Borwein stepsize, which achieves linear convergence for various convex functions and outperforms or matches existing methods in numerical experiments on standard datasets.
Variable metric proximal gradient methods with different metric selections have been widely used in composite optimization. Combining the Barzilai-Borwein (BB) method with a diagonal selection strategy for the metric, the diagonal BB stepsize can keep low per-step computation cost as the scalar BB stepsize and better capture the local geometry of the problem. In this paper, we propose a variable metric mini-batch proximal stochastic recursive gradient algorithm VM-mSRGBB, which updates the metric using a new diagonal BB stepsize. The linear convergence of VM-mSRGBB is established for strongly convex, non-strongly convex and convex functions. Numerical experiments on standard data sets show that VM-mSRGBB is better than or comparable to some variance reduced stochastic gradient methods with best-tuned scalar stepsizes or BB stepsizes. Furthermore, the performance of VM-mSRGBB is superior to some advanced mini-batch proximal stochastic gradient methods.