On the convergence of stochastic variance reduced gradient for linear inverse problems
This work provides theoretical guarantees for SVRG in inverse problems, which is incremental as it extends existing analysis to a specific domain.
The authors tackled the convergence of stochastic variance reduced gradient (SVRG) methods for solving linear inverse problems in Hilbert spaces, proving that regularized SVRG achieves optimal convergence rates without early stopping rules and standard SVRG is optimal for nonsmooth solutions with a priori stopping rules.
Stochastic variance reduced gradient (SVRG) is an accelerated version of stochastic gradient descent based on variance reduction, and is promising for solving large-scale inverse problems. In this work, we analyze SVRG and a regularized version that incorporates a priori knowledge of the problem, for solving linear inverse problems in Hilbert spaces. We prove that, with suitable constant step size schedules and regularity conditions, the regularized SVRG can achieve optimal convergence rates in terms of the noise level without any early stopping rules, provided that the truncation level is chosen suitably, and standard SVRG is also optimal for problems with nonsmooth solutions under a priori stopping rules. The analysis is based on an explicit error recursion and suitable a priori estimates on the inner loop updates with respect to the anchor point. Numerical experiments are provided to complement the theoretical analysis.