Generalization Performance of Empirical Risk Minimization on Over-parameterized Deep ReLU Nets
This addresses the theoretical understanding of generalization in deep learning for researchers, providing rigorous proofs but is incremental in nature.
The paper tackles the generalization performance of empirical risk minimization on over-parameterized deep ReLU networks by proving that perfect global minima achieve almost optimal generalization error bounds for various data types under mild conditions, filling a gap between optimization and generalization.
In this paper, we study the generalization performance of global minima for implementing empirical risk minimization (ERM) on over-parameterized deep ReLU nets. Using a novel deepening scheme for deep ReLU nets, we rigorously prove that there exist perfect global minima achieving almost optimal generalization error bounds for numerous types of data under mild conditions. Since over-parameterization is crucial to guarantee that the global minima of ERM on deep ReLU nets can be realized by the widely used stochastic gradient descent (SGD) algorithm, our results indeed fill a gap between optimization and generalization.