Nonparametric Feature Impact and Importance
This addresses a critical issue for data scientists and practitioners in making accurate interpretations, though it is incremental as it builds on existing nonparametric methods.
The paper tackles the problem of practitioners conflating feature importance with feature impact, which can lead to misinterpretations in fields like business or medicine, by proposing mathematical definitions derived from partial dependence curves that operate directly on data, showing competitive ranking with existing feature selection techniques on three real datasets.
Practitioners use feature importance to rank and eliminate weak predictors during model development in an effort to simplify models and improve generality. Unfortunately, they also routinely conflate such feature importance measures with feature impact, the isolated effect of an explanatory variable on the response variable. This can lead to real-world consequences when importance is inappropriately interpreted as impact for business or medical insight purposes. The dominant approach for computing importances is through interrogation of a fitted model, which works well for feature selection, but gives distorted measures of feature impact. The same method applied to the same data set can yield different feature importances, depending on the model, leading us to conclude that impact should be computed directly from the data. While there are nonparametric feature selection algorithms, they typically provide feature rankings, rather than measures of impact or importance. They also typically focus on single-variable associations with the response. In this paper, we give mathematical definitions of feature impact and importance, derived from partial dependence curves, that operate directly on the data. To assess quality, we show that features ranked by these definitions are competitive with existing feature selection techniques using three real data sets for predictive tasks.