On the Asymptotic Learning Curves of Kernel Ridge Regression under Power-law Decay
This addresses the theoretical understanding of overfitting in neural networks for researchers in statistical learning, though it is incremental by building on prior kernel regression work.
The paper rigorously characterizes the learning curves of kernel ridge regression under realistic assumptions, showing that benign overfitting in wide neural networks occurs only when noise levels are small.
The widely observed 'benign overfitting phenomenon' in the neural network literature raises the challenge to the 'bias-variance trade-off' doctrine in the statistical learning theory. Since the generalization ability of the 'lazy trained' over-parametrized neural network can be well approximated by that of the neural tangent kernel regression, the curve of the excess risk (namely, the learning curve) of kernel ridge regression attracts increasing attention recently. However, most recent arguments on the learning curve are heuristic and are based on the 'Gaussian design' assumption. In this paper, under mild and more realistic assumptions, we rigorously provide a full characterization of the learning curve: elaborating the effect and the interplay of the choice of the regularization parameter, the source condition and the noise. In particular, our results suggest that the 'benign overfitting phenomenon' exists in very wide neural networks only when the noise level is small.