Randomization Can Reduce Both Bias and Variance: A Case Study in Random Forests
This provides incremental insights into the theoretical understanding of random forests for machine learning practitioners, enhancing tuning strategies.
The paper tackles the problem of understanding why random forests outperform bagging ensembles, particularly in high signal-to-noise ratio settings, by empirically demonstrating that random forests reduce both bias and variance, leading to improved performance.
We study the often overlooked phenomenon, first noted in \cite{breiman2001random}, that random forests appear to reduce bias compared to bagging. Motivated by an interesting paper by \cite{mentch2020randomization}, where the authors explain the success of random forests in low signal-to-noise ratio (SNR) settings through regularization, we explore how random forests can capture patterns in the data that bagging ensembles fail to capture. We empirically demonstrate that in the presence of such patterns, random forests reduce bias along with variance and can increasingly outperform bagging ensembles when SNR is high. Our observations offer insights into the real-world success of random forests across a range of SNRs and enhance our understanding of the difference between random forests and bagging ensembles. Our investigations also yield practical insights into the importance of tuning $mtry$ in random forests.