A Stochastic Variance Reduced Gradient using Barzilai-Borwein Techniques as Second Order Information
This is an incremental improvement to optimization algorithms for machine learning practitioners working with large-scale problems.
The authors tackled the problem of improving stochastic variance reduced gradient (SVRG) methods by incorporating curvature information from the Barzilai-Borwein (BB) technique to reduce gradient variance. They showed that their method with constant step size outperforms existing variance-reduced methods on some benchmark datasets.
In this paper, we consider to improve the stochastic variance reduce gradient (SVRG) method via incorporating the curvature information of the objective function. We propose to reduce the variance of stochastic gradients using the computationally efficient Barzilai-Borwein (BB) method by incorporating it into the SVRG. We also incorporate a BB-step size as its variant. We prove its linear convergence theorem that works not only for the proposed method but also for the other existing variants of SVRG with second-order information. We conduct the numerical experiments on the benchmark datasets and show that the proposed method with constant step size performs better than the existing variance reduced methods for some test problems.