The Dimension Strikes Back with Gradients: Generalization of Gradient Methods in Stochastic Convex Optimization
This resolves open questions about the unavoidable dimension dependence in generalization for gradient methods, impacting theoretical machine learning.
The paper tackles the dimension dependence of generalization in stochastic convex optimization, showing that gradient descent requires at least Ω(√d) training examples to achieve non-trivial test error, and similarly for stochastic gradient descent to reach non-trivial empirical error, answering open questions from prior work.
We study the generalization performance of gradient methods in the fundamental stochastic convex optimization setting, focusing on its dimension dependence. First, for full-batch gradient descent (GD) we give a construction of a learning problem in dimension $d=O(n^2)$, where the canonical version of GD (tuned for optimal performance of the empirical risk) trained with $n$ training examples converges, with constant probability, to an approximate empirical risk minimizer with $Ω(1)$ population excess risk. Our bound translates to a lower bound of $Ω(\sqrt{d})$ on the number of training examples required for standard GD to reach a non-trivial test error, answering an open question raised by Feldman (2016) and Amir, Koren, and Livni (2021b) and showing that a non-trivial dimension dependence is unavoidable. Furthermore, for standard one-pass stochastic gradient descent (SGD), we show that an application of the same construction technique provides a similar $Ω(\sqrt{d})$ lower bound for the sample complexity of SGD to reach a non-trivial empirical error, despite achieving optimal test performance. This again provides an exponential improvement in the dimension dependence compared to previous work (Koren, Livni, Mansour, and Sherman, 2022), resolving an open question left therein.