Lassoed Tree Boosting
This provides a theoretical foundation for gradient boosting's effectiveness in high-dimensional settings, addressing a key bottleneck for practitioners in machine learning.
The paper tackles the problem of establishing convergence rates for gradient boosting in high-dimensional nonparametric spaces, proving that a lassoed gradient boosted tree algorithm with early stopping achieves faster than n^{-1/4} L2 convergence, independent of dimension, sparsity, or smoothness, with simulations and real data confirming performance on par with standard boosting.
Gradient boosting performs exceptionally in most prediction problems and scales well to large datasets. In this paper we prove that a ``lassoed'' gradient boosted tree algorithm with early stopping achieves faster than $n^{-1/4}$ L2 convergence in the large nonparametric space of cadlag functions of bounded sectional variation. This rate is remarkable because it does not depend on the dimension, sparsity, or smoothness. We use simulation and real data to confirm our theory and demonstrate empirical performance and scalability on par with standard boosting. Our convergence proofs are based on a novel, general theorem on early stopping with empirical loss minimizers of nested Donsker classes.