Many Faces of Feature Importance: Comparing Built-in and Post-hoc Feature Importance in Text Classification
This work addresses the consistency of feature importance methods for researchers and practitioners in explainable AI, but it is incremental as it compares existing methods without introducing new ones.
The study systematically compares feature importance methods in text classification, finding that traditional models produce more similar important features than deep learning models, and post-hoc methods yield more similar features between models than built-in methods, with similarity varying across instances and not correlating with prediction agreement.
Feature importance is commonly used to explain machine predictions. While feature importance can be derived from a machine learning model with a variety of methods, the consistency of feature importance via different methods remains understudied. In this work, we systematically compare feature importance from built-in mechanisms in a model such as attention values and post-hoc methods that approximate model behavior such as LIME. Using text classification as a testbed, we find that 1) no matter which method we use, important features from traditional models such as SVM and XGBoost are more similar with each other, than with deep learning models; 2) post-hoc methods tend to generate more similar important features for two models than built-in methods. We further demonstrate how such similarity varies across instances. Notably, important features do not always resemble each other better when two models agree on the predicted label than when they disagree.