On the Role of Initialization on the Implicit Bias in Deep Linear Networks
This work addresses a theoretical gap in deep learning by clarifying the role of initialization in optimization and generalization paradoxes, which is incremental to existing research on implicit bias sources.
The study investigates how weight initialization influences the implicit bias in deep linear networks, focusing on solving underdetermined linear systems to explain why these networks generalize well despite fitting data perfectly.
Despite Deep Learning's (DL) empirical success, our theoretical understanding of its efficacy remains limited. One notable paradox is that while conventional wisdom discourages perfect data fitting, deep neural networks are designed to do just that, yet they generalize effectively. This study focuses on exploring this phenomenon attributed to the implicit bias at play. Various sources of implicit bias have been identified, such as step size, weight initialization, optimization algorithm, and number of parameters. In this work, we focus on investigating the implicit bias originating from weight initialization. To this end, we examine the problem of solving underdetermined linear systems in various contexts, scrutinizing the impact of initialization on the implicit regularization when using deep networks to solve such systems. Our findings elucidate the role of initialization in the optimization and generalization paradoxes, contributing to a more comprehensive understanding of DL's performance characteristics.