Improved Learning Rates for Stochastic Optimization: Two Theoretical Viewpoints
This work provides incremental theoretical improvements for machine learning researchers by offering state-of-the-art learning rates under weaker assumptions, enhancing the understanding of generalization performance in stochastic optimization.
The paper tackles the problem of improving learning rates for stochastic optimization methods like empirical risk minimization (ERM) and stochastic gradient descent (SGD), achieving high probability rates of O(1/n) under milder assumptions in convex learning and similar rates in nonconvex learning, with faster O(1/n^2) rates in uniform convergence regimes.
Generalization performance of stochastic optimization stands a central place in learning theory. In this paper, we investigate the excess risk performance and towards improved learning rates for two popular approaches of stochastic optimization: empirical risk minimization (ERM) and stochastic gradient descent (SGD). Although there exists plentiful generalization analysis of ERM and SGD for supervised learning, current theoretical understandings of ERM and SGD either have stronger assumptions in convex learning, e.g., strong convexity, or show slow rates and less studied in nonconvex learning. Motivated by these problems, we aim to provide improved rates under milder assumptions in convex learning and derive faster rates in nonconvex learning. It is notable that our analysis span two popular theoretical viewpoints: \emph{stability} and \emph{uniform convergence}. Specifically, in stability regime, we present high probability learning rates of order $\mathcal{O} (1/n)$ w.r.t. the sample size $n$ for ERM and SGD with milder assumptions in convex learning and similar high probability rates of order $\mathcal{O} (1/n)$ in nonconvex learning, rather than in expectation. Furthermore, this type of learning rate is improved to faster order $\mathcal{O} (1/n^2)$ in uniform convergence regime. To our best knowledge, for ERM and SGD, the learning rates presented in this paper are all state-of-the-art.