Learning Landmark-Based Ensembles with Random Fourier Features and Gradient Boosting
This work addresses the challenge of kernel selection in machine learning for practitioners, offering an incremental improvement over existing Multiple Kernel Learning and Boosting-based methods.
The authors tackled the problem of learning task-adapted kernel ensembles by proposing a Gradient Boosting algorithm that optimizes Random Fourier Features barycenters, resulting in a more versatile and easier-to-setup method with better performance compared to state-of-the-art techniques.
We propose a Gradient Boosting algorithm for learning an ensemble of kernel functions adapted to the task at hand. Unlike state-of-the-art Multiple Kernel Learning techniques that make use of a pre-computed dictionary of kernel functions to select from, at each iteration we fit a kernel by approximating it as a weighted sum of Random Fourier Features (RFF) and by optimizing their barycenter. This allows us to obtain a more versatile method, easier to setup and likely to have better performance. Our study builds on a recent result showing one can learn a kernel from RFF by computing the minimum of a PAC-Bayesian bound on the kernel alignment generalization loss, which is obtained efficiently from a closed-form solution. We conduct an experimental analysis to highlight the advantages of our method w.r.t. both Boosting-based and kernel-learning state-of-the-art methods.