A Unified Convergence Theorem for Stochastic Optimization Methods
This work provides a foundational tool for analyzing convergence in stochastic optimization, benefiting researchers and practitioners in machine learning and optimization, though it is incremental as it builds on existing methods.
The authors developed a unified convergence theorem that simplifies deriving expected and almost sure convergence results for stochastic optimization methods, applying it to recover known results for SGD and RR under more general settings and establish new results for prox-SGD and SMM in nonsmooth nonconvex problems.
In this work, we provide a fundamental unified convergence theorem used for deriving expected and almost sure convergence results for a series of stochastic optimization methods. Our unified theorem only requires to verify several representative conditions and is not tailored to any specific algorithm. As a direct application, we recover expected and almost sure convergence results of the stochastic gradient method (SGD) and random reshuffling (RR) under more general settings. Moreover, we establish new expected and almost sure convergence results for the stochastic proximal gradient method (prox-SGD) and stochastic model-based methods (SMM) for nonsmooth nonconvex optimization problems. These applications reveal that our unified theorem provides a plugin-type convergence analysis and strong convergence guarantees for a wide class of stochastic optimization methods.