Cocoercivity, Smoothness and Bias in Variance-Reduced Stochastic Gradient Methods
This work addresses the analysis of biased updates in variance-reduced methods for optimization, which is incremental as it builds on existing SAG/SAGA methods.
The paper introduces SVAG, a variance-reduced stochastic gradient method with adjustable bias, to analyze biased updates in optimization, showing that step-size requirements differ when operators are gradients versus non-gradients, and numerical experiments indicate bias primarily affects early convergence stages.
With the purpose of examining biased updates in variance-reduced stochastic gradient methods, we introduce SVAG, a SAG/SAGA-like method with adjustable bias. SVAG is analyzed in a cocoercive root-finding setting, a setting which yields the same results as in the usual smooth convex optimization setting for the ordinary proximal-gradient method. We show that the same is not true for SVAG when biased updates are used. The step-size requirements for when the operators are gradients are significantly less restrictive compared to when they are not. This highlights the need to not rely solely on cocoercivity when analyzing variance-reduced methods meant for optimization. Our analysis either match or improve on previously known convergence conditions for SAG and SAGA. However, in the biased cases they still do not correspond well with practical experiences and we therefore examine the effect of bias numerically on a set of classification problems. The choice of bias seem to primarily affect the early stages of convergence and in most cases the differences vanish in the later stages of convergence. However, the effect of the bias choice is still significant in a couple of cases.