Exact Solutions of a Deep Linear Network
This work addresses a fundamental problem in understanding neural network optimization landscapes for researchers in machine learning theory, though it is incremental as it builds on prior studies of linear networks.
The paper derived exact analytical expressions for the global minima of deep linear networks with weight decay and stochastic neurons, revealing that weight decay can create problematic minima at zero in networks with more than one hidden layer, unlike single-layer networks. This finding indicates that standard initialization methods are generally inadequate for optimizing neural networks.
This work finds the analytical expression of the global minima of a deep linear network with weight decay and stochastic neurons, a fundamental model for understanding the landscape of neural networks. Our result implies that the origin is a special point in deep neural network loss landscape where highly nonlinear phenomenon emerges. We show that weight decay strongly interacts with the model architecture and can create bad minima at zero in a network with more than $1$ hidden layer, qualitatively different from a network with only $1$ hidden layer. Practically, our result implies that common deep learning initialization methods are insufficient to ease the optimization of neural networks in general.