Optimization Landscapes of Wide Deep Neural Networks Are Benign
This work addresses the optimization challenges in deep learning for researchers and practitioners, providing theoretical insights into why wide networks are easier to train, though it is incremental as it builds on existing beliefs.
The paper tackles the problem of understanding optimization difficulties in wide deep neural networks by showing that both constrained and unconstrained empirical-risk minimization over such networks have no confined points, substantiating the belief that wide networks are both highly expressive and comparably easy to optimize.
We analyze the optimization landscapes of deep learning with wide networks. We highlight the importance of constraints for such networks and show that constraint -- as well as unconstraint -- empirical-risk minimization over such networks has no confined points, that is, suboptimal parameters that are difficult to escape from. Hence, our theories substantiate the common belief that wide neural networks are not only highly expressive but also comparably easy to optimize.