Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retraining
This addresses the need for reliable explanation methods in NLP for researchers and practitioners, but it is incremental as it builds on existing faithfulness evaluation techniques.
The paper tackled the problem of evaluating how well importance measures in NLP models reflect the model's logic, known as faithfulness, by proposing Recursive ROAR, a new metric that recursively masks important tokens and retrains models, resulting in performance curves and a summarizing metric called RACU for comparison. They found that faithfulness is model-dependent and task-dependent, contradicting prior evaluations, with results tested on 4 importance measures across 8 datasets using LSTM-attention and RoBERTa models.
To explain NLP models a popular approach is to use importance measures, such as attention, which inform input tokens are important for making a prediction. However, an open question is how well these explanations accurately reflect a model's logic, a property called faithfulness. To answer this question, we propose Recursive ROAR, a new faithfulness metric. This works by recursively masking allegedly important tokens and then retraining the model. The principle is that this should result in worse model performance compared to masking random tokens. The result is a performance curve given a masking-ratio. Furthermore, we propose a summarizing metric using relative area-between-curves (RACU), which allows for easy comparison across papers, models, and tasks. We evaluate 4 different importance measures on 8 different datasets, using both LSTM-attention models and RoBERTa models. We find that the faithfulness of importance measures is both model-dependent and task-dependent. This conclusion contradicts previous evaluations in both computer vision and faithfulness of attention literature.