Optimization Insights into Deep Diagonal Linear Networks
This provides incremental theoretical insights into optimization dynamics for researchers in machine learning theory.
The paper tackles the limited theoretical understanding of overparameterized models by analyzing the implicit regularization of gradient flow in deep diagonal neural networks, showing it induces a mirror flow dynamic biased towards specific solutions based on initialization.
Overparameterized models trained with (stochastic) gradient descent are ubiquitous in modern machine learning. These large models achieve unprecedented performance on test data, but their theoretical understanding is still limited. In this paper, we take a step towards filling this gap by adopting an optimization perspective. More precisely, we study the implicit regularization properties of the gradient flow "algorithm" for estimating the parameters of a deep diagonal neural network. Our main contribution is showing that this gradient flow induces a mirror flow dynamic on the model, meaning that it is biased towards a specific solution of the problem depending on the initialization of the network. Along the way, we prove several properties of the trajectory.