Diagonal Linear Networks and the Lasso Regularization Path
This provides theoretical insights into neural network training dynamics for researchers in machine learning, though it is incremental as it builds on prior analysis of diagonal linear networks.
The paper tackles the problem of understanding the implicit regularization in diagonal linear networks by showing that their training trajectory closely approximates the lasso regularization path, with training time acting as an inverse regularization parameter, supported by rigorous results and simulations.
Diagonal linear networks are neural networks with linear activation and diagonal weight matrices. Their theoretical interest is that their implicit regularization can be rigorously analyzed: from a small initialization, the training of diagonal linear networks converges to the linear predictor with minimal 1-norm among minimizers of the training loss. In this paper, we deepen this analysis showing that the full training trajectory of diagonal linear networks is closely related to the lasso regularization path. In this connection, the training time plays the role of an inverse regularization parameter. Both rigorous results and simulations are provided to illustrate this conclusion. Under a monotonicity assumption on the lasso regularization path, the connection is exact while in the general case, we show an approximate connection.