Can a Single Tree Outperform an Entire Forest?
This work addresses the performance gap for users needing interpretable and lightweight models, though it is incremental as it builds on existing tree methods.
The study tackled the problem of single decision trees underperforming compared to random forests by developing a gradient-based optimization framework for oblique regression trees, resulting in an average 2.03% improvement in testing accuracy over classic random forests.
The prevailing mindset is that a single decision tree underperforms classic random forests in testing accuracy, despite its advantages in interpretability and lightweight structure. This study challenges such a mindset by significantly improving the testing accuracy of an oblique regression tree through our gradient-based entire tree optimization framework, making its performance comparable to the classic random forest. Our approach reformulates tree training as a differentiable unconstrained optimization task, employing a scaled sigmoid approximation strategy. To ameliorate numerical instability, we propose an algorithmic scheme that solves a sequence of increasingly accurate approximations. Additionally, a subtree polish strategy is implemented to reduce approximation errors accumulated across the tree. Extensive experiments on 16 datasets demonstrate that our optimized tree outperforms the classic random forest by an average of $2.03\%$ improvements in testing accuracy.