Effect of Depth and Width on Local Minima in Deep Learning
This addresses the problem of understanding optimization landscapes in deep learning for researchers, providing theoretical insights without incremental assumptions.
The paper analyzes how increasing depth and width in deep neural networks improves local minima quality towards the global optimum, without relying on over-parameterization assumptions, and shows that local minima values are no worse than optimal values in classical models, supported by empirical tests on synthetic, MNIST, CIFAR-10, and SVHN datasets.
In this paper, we analyze the effects of depth and width on the quality of local minima, without strong over-parameterization and simplification assumptions in the literature. Without any simplification assumption, for deep nonlinear neural networks with the squared loss, we theoretically show that the quality of local minima tends to improve towards the global minimum value as depth and width increase. Furthermore, with a locally-induced structure on deep nonlinear neural networks, the values of local minima of neural networks are theoretically proven to be no worse than the globally optimal values of corresponding classical machine learning models. We empirically support our theoretical observation with a synthetic dataset as well as MNIST, CIFAR-10 and SVHN datasets. When compared to previous studies with strong over-parameterization assumptions, the results in this paper do not require over-parameterization, and instead show the gradual effects of over-parameterization as consequences of general results.