A Unifying View on Implicit Bias in Training Linear Neural Networks
This work addresses the theoretical understanding of implicit bias for researchers in machine learning, providing a unifying framework that is incremental by extending and generalizing prior results.
The paper tackles the problem of understanding implicit bias in training linear neural networks by analyzing gradient flow, showing that for separable classification it finds a stationary point of an ℓ_{2/L} max-margin problem, and for underdetermined regression it minimizes a norm-like function interpolating between weighted ℓ₁ and ℓ₂ norms, with theorems subsuming existing results and removing convergence assumptions.
We study the implicit bias of gradient flow (i.e., gradient descent with infinitesimal step size) on linear neural network training. We propose a tensor formulation of neural networks that includes fully-connected, diagonal, and convolutional networks as special cases, and investigate the linear version of the formulation called linear tensor networks. With this formulation, we can characterize the convergence direction of the network parameters as singular vectors of a tensor defined by the network. For $L$-layer linear tensor networks that are orthogonally decomposable, we show that gradient flow on separable classification finds a stationary point of the $\ell_{2/L}$ max-margin problem in a "transformed" input space defined by the network. For underdetermined regression, we prove that gradient flow finds a global minimum which minimizes a norm-like function that interpolates between weighted $\ell_1$ and $\ell_2$ norms in the transformed input space. Our theorems subsume existing results in the literature while removing standard convergence assumptions. We also provide experiments that corroborate our analysis.