Uncertainty in Gradient Boosting via Ensembles
This work addresses uncertainty estimation for gradient boosting in high-risk applications, but it is incremental as it adapts ensemble methods to an existing model type.
The paper tackles the problem of quantifying predictive uncertainty in gradient boosting models, which are widely used for tabular data but under-explored for uncertainty estimation, and finds that ensembles of these models can detect anomalous inputs but have limited ability to improve total uncertainty, while proposing a virtual ensemble to reduce complexity.
For many practical, high-risk applications, it is essential to quantify uncertainty in a model's predictions to avoid costly mistakes. While predictive uncertainty is widely studied for neural networks, the topic seems to be under-explored for models based on gradient boosting. However, gradient boosting often achieves state-of-the-art results on tabular data. This work examines a probabilistic ensemble-based framework for deriving uncertainty estimates in the predictions of gradient boosting classification and regression models. We conducted experiments on a range of synthetic and real datasets and investigated the applicability of ensemble approaches to gradient boosting models that are themselves ensembles of decision trees. Our analysis shows that ensembles of gradient boosting models successfully detect anomalous inputs while having limited ability to improve the predicted total uncertainty. Importantly, we also propose a concept of a virtual ensemble to get the benefits of an ensemble via only one gradient boosting model, which significantly reduces complexity.