On the Gaussian process limit of Bayesian Additive Regression Trees
This work provides a novel analytical framework for understanding and developing BART and GP regression, with potential applications in nonparametric Bayesian regression, though it is incremental in linking existing methods.
The paper tackles the problem of connecting Bayesian Additive Regression Trees (BART) to Gaussian process (GP) regression by deriving the exact BART prior covariance function, enabling implementation of BART as GP regression. The result shows that this GP limit is initially worse than standard BART but becomes competitive after hyperparameter tuning, offering analytical likelihoods that simplify model building.
Bayesian Additive Regression Trees (BART) is a nonparametric Bayesian regression technique of rising fame. It is a sum-of-decision-trees model, and is in some sense the Bayesian version of boosting. In the limit of infinite trees, it becomes equivalent to Gaussian process (GP) regression. This limit is known but has not yet led to any useful analysis or application. For the first time, I derive and compute the exact BART prior covariance function. With it I implement the infinite trees limit of BART as GP regression. Through empirical tests, I show that this limit is worse than standard BART in a fixed configuration, but also that tuning its hyperparameters in the natural GP way makes it competitive with BART. The advantage of using a GP surrogate of BART is the analytical likelihood, which simplifies model building and sidesteps the complex BART MCMC algorithm. More generally, this study opens new ways to understand and develop BART and GP regression. The implementation of BART as GP is available in the Python package lsqfitgp.