Adaptivity for Regularized Kernel Methods by Lepskii's Principle
This work addresses the challenge of parameter selection in kernel methods for researchers and practitioners, offering an incremental improvement by adapting an existing principle to achieve optimality with theoretical guarantees.
The paper tackles the problem of adaptively choosing the regularization parameter in RKHS regression without prior knowledge of the regression function's structure, showing that a modified Lepskii Principle achieves minimax optimal adaptivity up to a log-log factor. It also demonstrates that balancing in the L² norm automatically yields optimal results in stronger norms, a convenience not proven for other methods like Hold-Out.
We address the problem of {\it adaptivity} in the framework of reproducing kernel Hilbert space (RKHS) regression. More precisely, we analyze estimators arising from a linear regularization scheme $g_\lam$. In practical applications, an important task is to choose the regularization parameter $\lam$ appropriately, i.e. based only on the given data and independently on unknown structural assumptions on the regression function. An attractive approach avoiding data-splitting is the {\it Lepskii Principle} (LP), also known as the {\it Balancing Principle} is this setting. We show that a modified parameter choice based on (LP) is minimax optimal adaptive, up to $\log\log(n)$. A convenient result is the fact that balancing in $L^2(ν)-$ norm, which is easiest, automatically gives optimal balancing in all stronger norms, interpolating between $L^2(ν)$ and the RKHS. An analogous result is open for other classical approaches to data dependent choices of the regularization parameter, e.g. for Hold-Out.