Convergence of gradient descent for deep neural networks
This provides theoretical guarantees for optimization convergence in non-overparameterized neural networks, addressing a fundamental problem in deep learning theory.
The paper establishes conditions for linear convergence of gradient descent in deep neural networks by introducing a local Polyak-Lojasiewicz criterion and verifying it for feedforward networks with smooth, strictly increasing activations, showing that a specific constructive initialization leads to provable convergence to zero training loss.
We give a simple local Polyak-Lojasiewicz (PL) criterion that guarantees linear (exponential) convergence of gradient flow and gradient descent to a zero-loss solution of a nonnegative objective. We then verify this criterion for the squared training loss of a feedforward neural network with smooth, strictly increasing activation functions, in a regime that is complementary to the usual over-parameterized analyses: the network width and depth are fixed, while the input data vectors are assumed to be linearly independent (in particular, the ambient input dimension is at least the number of data points). A notable feature of the verification is that it is constructive: it leads to a simple "positive" initialization (zero first-layer weights, strictly positive hidden-layer weights, and sufficiently large output-layer weights) under which gradient descent provably converges to an interpolating global minimizer of the training loss. We also discuss a probabilistic corollary for random initializations, clarify its dependence on the probability of the required initialization event, and provide numerical experiments showing that this theory-guided initialization can substantially accelerate optimization relative to standard random initializations at the same width.