A Robust Hypothesis Test for Tree Ensemble Pruning
This work addresses a theoretical gap in tree ensemble pruning for applied machine learning practitioners, offering a method that improves model performance with concrete results.
The paper tackles the lack of robust theoretical justifications for penalty terms in gradient boosted decision trees by developing a novel hypothesis test for split quality, which leads to a significant reduction in out-of-sample loss and provides a theoretically justified stopping condition for tree growing.
Gradient boosted decision trees are some of the most popular algorithms in applied machine learning. They are a flexible and powerful tool that can robustly fit to any tabular dataset in a scalable and computationally efficient way. One of the most critical parameters to tune when fitting these models are the various penalty terms used to distinguish signal from noise in the current model. These penalties are effective in practice, but are lacking in robust theoretical justifications. In this paper we develop and present a novel theoretically justified hypothesis test of split quality for gradient boosted tree ensembles and demonstrate that using this method instead of the common penalty terms leads to a significant reduction in out of sample loss. Additionally, this method provides a theoretically well-justified stopping condition for the tree growing algorithm. We also present several innovative extensions to the method, opening the door for a wide variety of novel tree pruning algorithms.