Generalizing Gain Penalization for Feature Selection in Tree-based Models
This work addresses feature selection for users of tree-based models, offering incremental improvements in regularization and flexibility.
The paper tackles the problem of sub-optimal feature selection in tree-based models due to insufficient regularization, especially with correlated features, by developing a new gain penalization method that improves out-of-sample performance, validated on simulated and real data.
We develop a new approach for feature selection via gain penalization in tree-based models. First, we show that previous methods do not perform sufficient regularization and often exhibit sub-optimal out-of-sample performance, especially when correlated features are present. Instead, we develop a new gain penalization idea that exhibits a general local-global regularization for tree-based models. The new method allows for more flexibility in the choice of feature-specific importance weights. We validate our method on both simulated and real data and implement itas an extension of the popular R package ranger.