Beyond Cross-Validation: Adaptive Parameter Selection for Kernel-Based Gradient Descents
This work addresses the problem of parameter selection for kernel-based gradient descent algorithms, which is significant for machine learning practitioners and researchers working with these algorithms.
The authors tackled the problem of parameter selection for kernel-based gradient descent algorithms, achieving the optimal generalization error bound. Their adaptive strategy adapts effectively to different kernels, target functions, and error metrics, outperforming existing methods.
This paper proposes a novel parameter selection strategy for kernel-based gradient descent (KGD) algorithms, integrating bias-variance analysis with the splitting method. We introduce the concept of empirical effective dimension to quantify iteration increments in KGD, deriving an adaptive parameter selection strategy that is implementable. Theoretical verifications are provided within the framework of learning theory. Utilizing the recently developed integral operator approach, we rigorously demonstrate that KGD, equipped with the proposed adaptive parameter selection strategy, achieves the optimal generalization error bound and adapts effectively to different kernels, target functions, and error metrics. Consequently, this strategy showcases significant advantages over existing parameter selection methods for KGD.