New Lower Bounds for Stochastic Non-Convex Optimization through Divergence Decomposition
This provides foundational theoretical insights for researchers in optimization, though it is incremental as it builds on existing lower bound analyses.
The paper tackles fundamental limits of first-order stochastic optimization for non-convex functions like Quasar-Convexity and Quadratic Growth, establishing new lower bounds on noisy gradient queries that are tight up to logarithmic factors and showing dimensional thresholds affect complexity.
We study fundamental limits of first-order stochastic optimization in a range of nonconvex settings, including L-smooth functions satisfying Quasar-Convexity (QC), Quadratic Growth (QG), and Restricted Secant Inequalities (RSI). While the convergence properties of standard algorithms are well-understood in deterministic regimes, significantly fewer results address the stochastic case, where only unbiased and noisy gradients are available. We establish new lower bounds on the number of noisy gradient queries to minimize these classes of functions, also showing that they are tight (up to a logarithmic factor) in all the relevant quantities characterizing each class. Our approach reformulates the optimization task as a function identification problem, leveraging divergence decomposition arguments to construct a challenging subclass that leads to sharp lower bounds. Furthermore, we present a specialized algorithm in the one-dimensional setting that achieves faster rates, suggesting that certain dimensional thresholds are intrinsic to the complexity of non-convex stochastic optimization.