Notes on Worst-case Inefficiency of Gradient Descent Even in R^2
This reveals a fundamental limitation of gradient descent for non-convex optimization, impacting researchers and practitioners in machine learning and optimization.
The paper tackles the problem of gradient descent's inefficiency in escaping saddle points for non-convex optimization, showing that it can take exponential time even in simple two-dimensional functions, with experimental verification.
Gradient descent is a popular algorithm in optimization, and its performance in convex settings is mostly well understood. In non-convex settings, it has been shown that gradient descent is able to escape saddle points asymptotically and converge to local minimizers [Lee et. al. 2016]. Recent studies also show a perturbed version of gradient descent is enough to escape saddle points efficiently [Jin et. al. 2015, Ge et. al. 2017]. In this paper we show a negative result: gradient descent may take exponential time to escape saddle points, with non-pathological two dimensional functions. While our focus is theoretical, we also conduct experiments verifying our theoretical result. Through our analysis we demonstrate that stochasticity is essential to escape saddle points efficiently.