Convergence of stochastic gradient descent schemes for Lojasiewicz-landscapes
This provides theoretical guarantees for SGD convergence in non-convex optimization, relevant for machine learning practitioners using neural networks, though it is incremental as it extends existing convergence results to broader conditions.
The paper proves that stochastic gradient descent (SGD) and its momentum variant converge under weak assumptions, specifically when the objective function has a countable number of critical points or satisfies Lojasiewicz inequalities, as in analytic functions. It applies this to neural networks with analytic activation functions, showing convergence when training data is compactly supported.
In this article, we consider convergence of stochastic gradient descent schemes (SGD), including momentum stochastic gradient descent (MSGD), under weak assumptions on the underlying landscape. More explicitly, we show that on the event that the SGD stays bounded we have convergence of the SGD if there is only a countable number of critical points or if the objective function satisfies Lojasiewicz-inequalities around all critical levels as all analytic functions do. In particular, we show that for neural networks with analytic activation function such as softplus, sigmoid and the hyperbolic tangent, SGD converges on the event of staying bounded, if the random variables modelling the signal and response in the training are compactly supported.