High-Dimensional Kernel Methods under Covariate Shift: Data-Dependent Implicit Regularization
This work addresses covariate shift in high-dimensional kernel methods, providing theoretical insights for machine learning practitioners, but it is incremental as it builds on existing re-weighting and regularization concepts.
The paper analyzes kernel ridge regression in high dimensions under covariate shifts, showing that importance re-weighting reduces variance and that bias behavior varies with regularization scales, with theoretical results derived through asymptotic expansions.
This paper studies kernel ridge regression in high dimensions under covariate shifts and analyzes the role of importance re-weighting. We first derive the asymptotic expansion of high dimensional kernels under covariate shifts. By a bias-variance decomposition, we theoretically demonstrate that the re-weighting strategy allows for decreasing the variance. For bias, we analyze the regularization of the arbitrary or well-chosen scale, showing that the bias can behave very differently under different regularization scales. In our analysis, the bias and variance can be characterized by the spectral decay of a data-dependent regularized kernel: the original kernel matrix associated with an additional re-weighting matrix, and thus the re-weighting strategy can be regarded as a data-dependent regularization for better understanding. Besides, our analysis provides asymptotic expansion of kernel functions/vectors under covariate shift, which has its own interest.