Kernel regression in high dimensions: Refined analysis beyond double descent
This refines the double descent theory for kernel regression in high dimensions, addressing a theoretical bottleneck in understanding generalization across under- and over-parameterized regimes.
The paper provides a precise characterization of generalization in high-dimensional kernel ridge regression, showing that the risk curve can exhibit double-descent-like, bell-shaped, or monotonic behavior depending on data eigen-profiles and regularization, with experiments on synthetic and real data supporting these findings.
In this paper, we provide a precise characterization of generalization properties of high dimensional kernel ridge regression across the under- and over-parameterized regimes, depending on whether the number of training data n exceeds the feature dimension d. By establishing a bias-variance decomposition of the expected excess risk, we show that, while the bias is (almost) independent of d and monotonically decreases with n, the variance depends on n, d and can be unimodal or monotonically decreasing under different regularization schemes. Our refined analysis goes beyond the double descent theory by showing that, depending on the data eigen-profile and the level of regularization, the kernel regression risk curve can be a double-descent-like, bell-shaped, or monotonic function of n. Experiments on synthetic and real data are conducted to support our theoretical findings.