Adaptive Forests For Classification
This addresses the need for more accurate classification models in machine learning applications, though it appears incremental as it builds on existing tree-based methods.
The paper tackles the problem of improving classification performance by proposing Adaptive Forests (AF), which adaptively selects weights for CART trees, resulting in consistent outperformance over Random Forests, XGBoost, and other weighted methods across 20+ real-world datasets.
Random Forests (RF) and Extreme Gradient Boosting (XGBoost) are two of the most widely used and highly performing classification and regression models. They aggregate equally weighted CART trees, generated randomly in RF or sequentially in XGBoost. In this paper, we propose Adaptive Forests (AF), a novel approach that adaptively selects the weights of the underlying CART models. AF combines (a) the Optimal Predictive-Policy Trees (OP2T) framework to prescribe tailored, input-dependent unequal weights to trees and (b) Mixed Integer Optimization (MIO) to refine weight candidates dynamically, enhancing overall performance. We demonstrate that AF consistently outperforms RF, XGBoost, and other weighted RF in binary and multi-class classification problems over 20+ real-world datasets.