Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs
This provides theoretical insights into implicit bias in neural network training for a fundamental regression task, though it is incremental by extending analysis to correlated data.
The paper tackles the problem of learning a single neuron using a shallow ReLU network, proving that gradient flow from small initialization converges to zero loss and is implicitly biased to minimize parameter rank for correlated inputs, complementing prior work on orthogonal datasets.
We prove that, for the fundamental regression task of learning a single neuron, training a one-hidden layer ReLU network of any width by gradient flow from a small initialisation converges to zero loss and is implicitly biased to minimise the rank of network parameters. By assuming that the training points are correlated with the teacher neuron, we complement previous work that considered orthogonal datasets. Our results are based on a detailed non-asymptotic analysis of the dynamics of each hidden neuron throughout the training. We also show and characterise a surprising distinction in this setting between interpolator networks of minimal rank and those of minimal Euclidean norm. Finally we perform a range of numerical experiments, which corroborate our theoretical findings.