Total stability of kernel methods
This work addresses robustness in kernel-based learning for scenarios where hyperparameters or data-dependent kernels are approximated, which is incremental as it extends classical robust statistics to more general perturbations.
The paper investigates the stability of kernel methods when the probability measure, regularization parameter, and kernel are simultaneously perturbed, establishing conditions under which these methods remain stable for convex Lipschitz loss functions and bounded smooth kernels.
Regularized empirical risk minimization using kernels and their corresponding reproducing kernel Hilbert spaces (RKHSs) plays an important role in machine learning. However, the actually used kernel often depends on one or on a few hyperparameters or the kernel is even data dependent in a much more complicated manner. Examples are Gaussian RBF kernels, kernel learning, and hierarchical Gaussian kernels which were recently proposed for deep learning. Therefore, the actually used kernel is often computed by a grid search or in an iterative manner and can often only be considered as an approximation to the "ideal" or "optimal" kernel. The paper gives conditions under which classical kernel based methods based on a convex Lipschitz loss function and on a bounded and smooth kernel are stable, if the probability measure $P$, the regularization parameter $λ$, and the kernel $k$ may slightly change in a simultaneous manner. Similar results are also given for pairwise learning. Therefore, the topic of this paper is somewhat more general than in classical robust statistics, where usually only the influence of small perturbations of the probability measure $P$ on the estimated function is considered.