Global optimality conditions for deep neural networks
This provides theoretical insights into the error landscape for researchers in optimization and deep learning, though it is incremental as it builds on prior work on nonconvex optimization in neural networks.
The paper tackles the nonconvex optimization problem in deep neural networks by deriving necessary and sufficient conditions for critical points to be global minima in deep linear networks, with an efficiently checkable test, and extends similar sufficient conditions to nonlinear networks in a limited setting.
We study the error landscape of deep linear and nonlinear neural networks with the squared error loss. Minimizing the loss of a deep linear neural network is a nonconvex problem, and despite recent progress, our understanding of this loss surface is still incomplete. For deep linear networks, we present necessary and sufficient conditions for a critical point of the risk function to be a global minimum. Surprisingly, our conditions provide an efficiently checkable test for global optimality, while such tests are typically intractable in nonconvex optimization. We further extend these results to deep nonlinear neural networks and prove similar sufficient conditions for global optimality, albeit in a more limited function space setting.