Generalization Ability of Wide Residual Networks
This work addresses theoretical understanding of generalization in deep learning for researchers, providing insights into the conditions under which wide residual networks generalize well or poorly.
The paper analyzes the generalization ability of wide residual networks, showing that as width increases, the network's kernel converges to the residual neural tangent kernel, leading to generalization error matching kernel regression and achieving minimax rates with early stopping, but poor generalization if trained to overfit.
In this paper, we study the generalization ability of the wide residual network on $\mathbb{S}^{d-1}$ with the ReLU activation function. We first show that as the width $m\rightarrow\infty$, the residual network kernel (RNK) uniformly converges to the residual neural tangent kernel (RNTK). This uniform convergence further guarantees that the generalization error of the residual network converges to that of the kernel regression with respect to the RNTK. As direct corollaries, we then show $i)$ the wide residual network with the early stopping strategy can achieve the minimax rate provided that the target regression function falls in the reproducing kernel Hilbert space (RKHS) associated with the RNTK; $ii)$ the wide residual network can not generalize well if it is trained till overfitting the data. We finally illustrate some experiments to reconcile the contradiction between our theoretical result and the widely observed ``benign overfitting phenomenon''