A Geometric Approach of Gradient Descent Algorithms in Linear Neural Networks
This provides theoretical guarantees for gradient descent in linear neural networks, addressing a foundational issue in optimization for machine learning, though it is incremental as it builds on known empirical observations.
The authors tackled the problem of understanding gradient descent convergence in linear neural networks, proving that for single-hidden-layer networks, gradient descent trajectories converge to a global minimum under generic conditions, implying equivalence to least-squares estimation.
In this paper, we propose a geometric framework to analyze the convergence properties of gradient descent trajectories in the context of linear neural networks. We translate a well-known empirical observation of linear neural nets into a conjecture that we call the \emph{overfitting conjecture} which states that, for almost all training data and initial conditions, the trajectory of the corresponding gradient descent system converges to a global minimum. This would imply that the solution achieved by vanilla gradient descent algorithms is equivalent to that of the least-squares estimation, for linear neural networks of an arbitrary number of hidden layers. Built upon a key invariance property induced by the network structure, we first establish convergence of gradient descent trajectories to critical points of the square loss function in the case of linear networks of arbitrary depth. Our second result is the proof of the \emph{overfitting conjecture} in the case of single-hidden-layer linear networks with an argument based on the notion of normal hyperbolicity and under a generic property on the training data (i.e., holding for almost all training data).