Asymptotic Normality of Infinite Centered Random Forests -Application to Imbalanced Classification
This work addresses imbalanced data classification, a common problem in machine learning, by providing theoretical guarantees for random forests, though it is incremental as it builds on existing CRF methods.
The paper tackles imbalanced classification by theoretically analyzing Centered Random Forests (CRF) trained on rebalanced datasets, proving a Central Limit Theorem (CLT) with explicit rates and constants, and showing that a debiased estimator (IS-ICRF) reduces variance by up to 50% in high-imbalance settings.
Many classification tasks involve imbalanced data, in which a class is largely underrepresented. Several techniques consists in creating a rebalanced dataset on which a classifier is trained. In this paper, we study theoretically such a procedure, when the classifier is a Centered Random Forests (CRF). We establish a Central Limit Theorem (CLT) on the infinite CRF with explicit rates and exact constant. We then prove that the CRF trained on the rebalanced dataset exhibits a bias, which can be removed with appropriate techniques. Based on an importance sampling (IS) approach, the resulting debiased estimator, called IS-ICRF, satisfies a CLT centered at the prediction function value. For high imbalance settings, we prove that the IS-ICRF estimator enjoys a variance reduction compared to the ICRF trained on the original data. Therefore, our theoretical analysis highlights the benefits of training random forests on a rebalanced dataset (followed by a debiasing procedure) compared to using the original data. Our theoretical results, especially the variance rates and the variance reduction, appear to be valid for Breiman's random forests in our experiments.