Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy
This addresses theoretical understanding of implicit bias in machine learning, but it is incremental as it builds on existing models and focuses on asymptotic analysis.
The paper investigates how initialization scale and training accuracy affect implicit bias in deep linear classification, revealing that limit behaviors of gradient descent only emerge at extremely high training accuracies beyond 10^{-100}, and the bias at practical scales is more complex.
We provide a detailed asymptotic study of gradient flow trajectories and their implicit optimization bias when minimizing the exponential loss over "diagonal linear networks". This is the simplest model displaying a transition between "kernel" and non-kernel ("rich" or "active") regimes. We show how the transition is controlled by the relationship between the initialization scale and how accurately we minimize the training loss. Our results indicate that some limit behaviors of gradient descent only kick in at ridiculous training accuracies (well beyond $10^{-100}$). Moreover, the implicit bias at reasonable initialization scales and training accuracies is more complex and not captured by these limits.