Effects of Initialization Biases on Deep Neural Network Training Dynamics
This addresses a problem for machine learning practitioners by highlighting an incremental interaction between initialization biases and training components, which could impact model robustness and performance.
The paper investigates how Initial Guessing Bias, where untrained neural networks favor a few classes after random initialization, affects early training dynamics, finding that loss functions like Blurry and Piecewise-zero loss can fail to steer training effectively due to this bias.
Untrained large neural networks, just after random initialization, tend to favour a small subset of classes, assigning high predicted probabilities to these few classes and approximately zero probability to all others. This bias, termed Initial Guessing Bias, affects the early training dynamics, when the model is fitting to the coarse structure of the data. The choice of loss function against which to train the model has a large impact on how these early dynamics play out. Two recent loss functions, Blurry and Piecewise-zero loss, were designed for robustness to label errors but can become unable to steer the direction of training when exposed to this initial bias. Results indicate that the choice of loss function has a dramatic effect on the early phase training of networks, and highlights the need for careful consideration of how Initial Guessing Bias may interact with various components of the training scheme.