The Role of Depth, Width, and Tree Size in Expressiveness of Deep Forest
It provides foundational theoretical insights for researchers in machine learning, addressing a gap in understanding deep forest algorithms.
This work tackled the lack of theoretical understanding of how depth, width, and tree size affect the expressiveness of deep forests, showing that depth exponentially enhances expressiveness compared to the other parameters.
Random forests are classical ensemble algorithms that construct multiple randomized decision trees and aggregate their predictions using naive averaging. \citet{zhou2019deep} further propose a deep forest algorithm with multi-layer forests, which outperforms random forests in various tasks. The performance of deep forests is related to three hyperparameters in practice: depth, width, and tree size, but little has been known about its theoretical explanation. This work provides the first upper and lower bounds on the approximation complexity of deep forests concerning the three hyperparameters. Our results confirm the distinctive role of depth, which can exponentially enhance the expressiveness of deep forests compared with width and tree size. Experiments confirm the theoretical findings.