A Numerical Transform of Random Forest Regressors corrects Systematically-Biased Predictions
This addresses a specific issue in random forest regression for practitioners, offering a correction method that enhances prediction accuracy, though it is incremental as it builds on existing models.
The authors identified a systematic bias in random forest regression models across real-world and synthetic datasets, and developed a numerical transformation using training data to fully correct this bias, resulting in improved predictions in all evaluated datasets.
Over the past decade, random forest models have become widely used as a robust method for high-dimensional data regression tasks. In part, the popularity of these models arises from the fact that they require little hyperparameter tuning and are not very susceptible to overfitting. Random forest regression models are comprised of an ensemble of decision trees that independently predict the value of a (continuous) dependent variable; predictions from each of the trees are ultimately averaged to yield an overall predicted value from the forest. Using a suite of representative real-world datasets, we find a systematic bias in predictions from random forest models. We find that this bias is recapitulated in simple synthetic datasets, regardless of whether or not they include irreducible error (noise) in the data, but that models employing boosting do not exhibit this bias. Here we demonstrate the basis for this problem, and we use the training data to define a numerical transformation that fully corrects it. Application of this transformation yields improved predictions in every one of the real-world and synthetic datasets evaluated in our study.