Lower Bounds for Smooth Nonconvex Finite-Sum Optimization
This work addresses a gap in optimization theory by providing foundational lower bounds for nonconvex finite-sum problems, which is crucial for algorithm design and analysis in machine learning and AI.
The paper tackles the problem of establishing lower bounds for smooth nonconvex finite-sum optimization, proving tight complexity bounds for finding ε-suboptimal points and ε-approximate stationary points, and shows that existing algorithms achieve optimal IFO complexity up to logarithmic factors.
Smooth finite-sum optimization has been widely studied in both convex and nonconvex settings. However, existing lower bounds for finite-sum optimization are mostly limited to the setting where each component function is (strongly) convex, while the lower bounds for nonconvex finite-sum optimization remain largely unsolved. In this paper, we study the lower bounds for smooth nonconvex finite-sum optimization, where the objective function is the average of $n$ nonconvex component functions. We prove tight lower bounds for the complexity of finding $ε$-suboptimal point and $ε$-approximate stationary point in different settings, for a wide regime of the smallest eigenvalue of the Hessian of the objective function (or each component function). Given our lower bounds, we can show that existing algorithms including KatyushaX (Allen-Zhu, 2018), Natasha (Allen-Zhu, 2017), RapGrad (Lan and Yang, 2018) and StagewiseKatyusha (Chen and Yang, 2018) have achieved optimal Incremental First-order Oracle (IFO) complexity (i.e., number of IFO calls) up to logarithm factors for nonconvex finite-sum optimization. We also point out potential ways to further improve these complexity results, in terms of making stronger assumptions or by a different convergence analysis.