KTBoost: Combined Kernel and Tree Boosting
This work addresses the need for improved predictive modeling in machine learning by combining discontinuous and continuous learners, though it is incremental as it builds on existing boosting methods.
The paper tackles the problem of learning functions with varying regularity by introducing KTBoost, a boosting algorithm that combines regression trees and RKHS regression functions, and empirically shows it significantly outperforms both tree and kernel boosting in predictive accuracy across multiple data sets.
We introduce a novel boosting algorithm called `KTBoost' which combines kernel boosting and tree boosting. In each boosting iteration, the algorithm adds either a regression tree or reproducing kernel Hilbert space (RKHS) regression function to the ensemble of base learners. Intuitively, the idea is that discontinuous trees and continuous RKHS regression functions complement each other, and that this combination allows for better learning of functions that have parts with varying degrees of regularity such as discontinuities and smooth parts. We empirically show that KTBoost significantly outperforms both tree and kernel boosting in terms of predictive accuracy in a comparison on a wide array of data sets.