Why Learning of Large-Scale Neural Networks Behaves Like Convex Optimization
This provides a theoretical foundation for understanding optimization in deep learning, addressing a key problem for researchers and practitioners by showing that non-convex neural network training can behave like convex optimization.
The paper explains why gradient descent successfully trains large neural networks by proving that the objective functions become convex in a canonical model space, and shows that gradient descent converges to a global minimum with zero loss under full-rank conditions, which hold with high probability in over-parameterized networks.
In this paper, we present some theoretical work to explain why simple gradient descent methods are so successful in solving non-convex optimization problems in learning large-scale neural networks (NN). After introducing a mathematical tool called canonical space, we have proved that the objective functions in learning NNs are convex in the canonical model space. We further elucidate that the gradients between the original NN model space and the canonical space are related by a pointwise linear transformation, which is represented by the so-called disparity matrix. Furthermore, we have proved that gradient descent methods surely converge to a global minimum of zero loss provided that the disparity matrices maintain full rank. If this full-rank condition holds, the learning of NNs behaves in the same way as normal convex optimization. At last, we have shown that the chance to have singular disparity matrices is extremely slim in large NNs. In particular, when over-parameterized NNs are randomly initialized, the gradient decent algorithms converge to a global minimum of zero loss in probability.