Explanation Regularisation through the Lens of Attributions
This work challenges assumptions in interpretable AI for NLP, showing incremental insights by questioning the effectiveness of a popular method.
The paper investigates explanation regularisation (ER) for text classifiers, which aims to align model predictions with human-annotated rationales to improve out-of-domain performance, but finds that the claimed link between ER and reliance on plausible features is overstated and not the cause of OOD improvements.
Explanation regularisation (ER) has been introduced as a way to guide text classifiers to form their predictions relying on input tokens that humans consider plausible. This is achieved by introducing an auxiliary explanation loss that measures how well the output of an input attribution technique for the model agrees with human-annotated rationales. The guidance appears to benefit performance in out-of-domain (OOD) settings, presumably due to an increased reliance on "plausible" tokens. However, previous work has under-explored the impact of guidance on that reliance, particularly when reliance is measured using attribution techniques different from those used to guide the model. In this work, we seek to close this gap, and also explore the relationship between reliance on plausible features and OOD performance. We find that the connection between ER and the ability of a classifier to rely on plausible features has been overstated and that a stronger reliance on plausible tokens does not seem to be the cause for OOD improvements.