On Tighter Generalization Bound for Deep Neural Networks: CNNs, ResNets, and Beyond
This work provides incremental theoretical improvements for researchers in machine learning theory, offering tighter bounds for CNNs, ResNets, and general feedforward networks.
The authors tackled the problem of deriving tighter generalization error bounds for deep neural networks, achieving significantly improved bounds in terms of depth, width, and Jacobian properties, with numerical evaluation supporting the theory.
We establish a margin based data dependent generalization error bound for a general family of deep neural networks in terms of the depth and width, as well as the Jacobian of the networks. Through introducing a new characterization of the Lipschitz properties of neural network family, we achieve significantly tighter generalization bounds than existing results. Moreover, we show that the generalization bound can be further improved for bounded losses. Aside from the general feedforward deep neural networks, our results can be applied to derive new bounds for popular architectures, including convolutional neural networks (CNNs) and residual networks (ResNets). When achieving same generalization errors with previous arts, our bounds allow for the choice of larger parameter spaces of weight matrices, inducing potentially stronger expressive ability for neural networks. Numerical evaluation is also provided to support our theory.