Incremental Learning in Diagonal Linear Networks
This work provides incremental theoretical insights into a toy model of neural networks, primarily relevant for researchers studying optimization and regularization in machine learning.
The paper tackles the problem of understanding gradient flow in diagonal linear networks (DLNs) by analyzing their trajectory under small initialization, showing that incremental learning occurs with coordinates activated successively and sparse regularization decreasing over time, specifically in an underparametrized regime with anti-correlated features.
Diagonal linear networks (DLNs) are a toy simplification of artificial neural networks; they consist in a quadratic reparametrization of linear regression inducing a sparse implicit regularization. In this paper, we describe the trajectory of the gradient flow of DLNs in the limit of small initialization. We show that incremental learning is effectively performed in the limit: coordinates are successively activated, while the iterate is the minimizer of the loss constrained to have support on the active coordinates only. This shows that the sparse implicit regularization of DLNs decreases with time. This work is restricted to the underparametrized regime with anti-correlated features for technical reasons.