Opening the random forest black box by the analysis of the mutual impact of features
This work addresses feature selection and relationship analysis in high-dimensional data for researchers and practitioners using random forests, but it is incremental as it builds on existing variable importance measures.
The authors tackled the problem of understanding complex feature relationships in random forests by proposing two new methods, Mutual Forest Impact (MFI) and Mutual Impurity Reduction (MIR), which analyze mutual feature impacts and provide p-values for selection, showing promising results in simulations without common biases.
Random forest is a popular machine learning approach for the analysis of high-dimensional data because it is flexible and provides variable importance measures for the selection of relevant features. However, the complex relationships between the features are usually not considered for the selection and thus also neglected for the characterization of the analysed samples. Here we propose two novel approaches that focus on the mutual impact of features in random forests. Mutual forest impact (MFI) is a relation parameter that evaluates the mutual association of the featurs to the outcome and, hence, goes beyond the analysis of correlation coefficients. Mutual impurity reduction (MIR) is an importance measure that combines this relation parameter with the importance of the individual features. MIR and MFI are implemented together with testing procedures that generate p-values for the selection of related and important features. Applications to various simulated data sets and the comparison to other methods for feature selection and relation analysis show that MFI and MIR are very promising to shed light on the complex relationships between features and outcome. In addition, they are not affected by common biases, e.g. that features with many possible splits or high minor allele frequencies are prefered.