When Bias Helps Learning: Bridging Initial Prejudice and Trainability
This work addresses a foundational issue in machine learning by linking initialization properties to trainability, offering insights for improving neural network training.
The paper tackles the problem of understanding how initialization biases in deep neural networks affect trainability, proving that a systematic bias toward specific classes at initialization optimizes learning efficiency.
Understanding the statistical properties of deep neural networks (DNNs) at initialization is crucial for elucidating both their trainability and the intrinsic architectural biases they encode prior to data exposure. Mean-field (MF) analyses have demonstrated that the parameter distribution in randomly initialized networks dictates whether gradients vanish or explode. Recent work has shown that untrained DNNs exhibit an initial-guessing bias (IGB), in which large regions of the input space are assigned to a single class. In this work, we provide a theoretical proof linking IGB to MF analyses, establishing that a network predisposition toward specific classes is intrinsically tied to the conditions for efficient learning. This connection leads to a counterintuitive conclusion: the initialization that optimizes trainability is systematically biased rather than neutral. We validate our theory through experiments across multiple architectures and datasets.