Wide Boosting
This addresses a specific bottleneck in Gradient Boosting for tabular ML problems, offering an incremental improvement for practitioners dealing with correlated outputs.
The paper tackles the limitation of Gradient Boosting in handling correlated multi-dimensional outputs and lack of data embeddings by introducing Wide Boosting, which inserts a matrix multiplication to increase output dimension, resulting in improved performance on multi-dimensional tasks and more useful embeddings for downstream tasks.
Gradient Boosting (GB) is a popular methodology used to solve prediction problems by minimizing a differentiable loss function, $L$. GB performs very well on tabular machine learning (ML) problems; however, as a pure ML solver it lacks the ability to fit models with probabilistic but correlated multi-dimensional outputs, for example, multiple correlated Bernoulli outputs. GB also does not form intermediate abstract data embeddings, one property of Deep Learning that gives greater flexibility and performance on other types of problems. This paper presents a simple adjustment to GB motivated in part by artificial neural networks. Specifically, our adjustment inserts a matrix multiplication between the output of a GB model and the loss, $L$. This allows the output of a GB model to have increased dimension prior to being fed into the loss and is thus ``wider'' than standard GB implementations. We call our method Wide Boosting (WB) and show that WB outperforms GB on mult-dimesional output tasks and that the embeddings generated by WB contain are more useful in downstream prediction tasks than GB output predictions alone.