Zero loss guarantees and explicit minimizers for generic overparametrized Deep Learning networks
This addresses the problem of understanding loss minimization in overparametrized networks for researchers, clarifying the dichotomy between underparametrized and overparametrized settings, though it is incremental in analyzing theoretical guarantees.
The paper establishes conditions for overparametrized deep learning networks to achieve zero loss in supervised learning with generic data, providing explicit minimizers without gradient descent, and shows that increasing depth can hinder gradient descent efficiency due to rank loss of the training Jacobian.
We determine sufficient conditions for overparametrized deep learning (DL) networks to guarantee the attainability of zero loss in the context of supervised learning, for the $\mathcal{L}^2$ cost and {\em generic} training data. We present an explicit construction of the zero loss minimizers without invoking gradient descent. On the other hand, we point out that increase of depth can deteriorate the efficiency of cost minimization using a gradient descent algorithm by analyzing the conditions for rank loss of the training Jacobian. Our results clarify key aspects on the dichotomy between zero loss reachability in underparametrized versus overparametrized DL.