From Kernel Machines to Ensemble Learning
This provides a principled framework for designing ensemble learning methods, potentially benefiting machine learning practitioners by improving efficiency and flexibility, though it appears incremental as it builds on known equivalences.
The paper tackles the problem of constructing ensemble learning methods directly from kernel methods, showing that boosting-like procedures can solve SVM optimization problems without translation, enabling new ensemble methods and leveraging fast linear SVM solvers.
Ensemble methods such as boosting combine multiple learners to obtain better prediction than could be obtained from any individual learner. Here we propose a principled framework for directly constructing ensemble learning methods from kernel methods. Unlike previous studies showing the equivalence between boosting and support vector machines (SVMs), which needs a translation procedure, we show that it is possible to design boosting-like procedure to solve the SVM optimization problems. In other words, it is possible to design ensemble methods directly from SVM without any middle procedure. This finding not only enables us to design new ensemble learning methods directly from kernel methods, but also makes it possible to take advantage of those highly-optimized fast linear SVM solvers for ensemble learning. We exemplify this framework for designing binary ensemble learning as well as a new multi-class ensemble learning methods. Experimental results demonstrate the flexibility and usefulness of the proposed framework.