Tackling benign nonconvexity with smoothing and stochastic gradients
This provides a theoretical foundation for understanding SGD's success in deep learning, addressing a key problem for researchers and practitioners in machine learning.
The paper tackles the theoretical gap in explaining why stochastic gradient descent (SGD) works well for non-convex optimization in machine learning, showing that perturbed SGD converges to a global minimum for a broad class of non-convex functions, such as those close to convex-like functions, with linear convergence in some cases.
Non-convex optimization problems are ubiquitous in machine learning, especially in Deep Learning. While such complex problems can often be successfully optimized in practice by using stochastic gradient descent (SGD), theoretical analysis cannot adequately explain this success. In particular, the standard analyses do not show global convergence of SGD on non-convex functions, and instead show convergence to stationary points (which can also be local minima or saddle points). We identify a broad class of nonconvex functions for which we can show that perturbed SGD (gradient descent perturbed by stochastic noise -- covering SGD as a special case) converges to a global minimum (or a neighborhood thereof), in contrast to gradient descent without noise that can get stuck in local minima far from a global solution. For example, on non-convex functions that are relatively close to a convex-like (strongly convex or PL) function we show that SGD can converge linearly to a global optimum.