Convergence Analysis of SGD under Expected Smoothness
This work provides a refined theoretical framework for analyzing SGD, which is incremental but important for researchers in optimization and machine learning.
The paper tackles the convergence analysis of Stochastic Gradient Descent (SGD) under the expected smoothness condition, deriving explicit convergence rates with residual errors for various step-size schedules.
Stochastic gradient descent (SGD) is the workhorse of large-scale learning, yet classical analyses rely on assumptions that can be either too strong (bounded variance) or too coarse (uniform noise). The expected smoothness (ES) condition has emerged as a flexible alternative that ties the second moment of stochastic gradients to the objective value and the full gradient. This paper presents a self-contained convergence analysis of SGD under ES. We (i) refine ES with interpretations and sampling-dependent constants; (ii) derive bounds of the expectation of squared full gradient norm; and (iii) prove $O(1/K)$ rates with explicit residual errors for various step-size schedules. All proofs are given in full detail in the appendix. Our treatment unifies and extends recent threads (Khaled and Richtárik, 2020; Umeda and Iiduka, 2025).