Saliency Maps Generation for Automatic Text Summarization
This work addresses the reliability of explainable AI methods for complex tasks like text summarization, showing that current saliency map approaches can be misleading and incremental in improving evaluation protocols.
The paper applied Layer-Wise Relevance Propagation to a sequence-to-sequence attention model for text summarization, revealing that saliency maps sometimes accurately capture input feature usage but often do not, highlighting the need for caution in accepting them as explanations.
Saliency map generation techniques are at the forefront of explainable AI literature for a broad range of machine learning applications. Our goal is to question the limits of these approaches on more complex tasks. In this paper we apply Layer-Wise Relevance Propagation (LRP) to a sequence-to-sequence attention model trained on a text summarization dataset. We obtain unexpected saliency maps and discuss the rightfulness of these "explanations". We argue that we need a quantitative way of testing the counterfactual case to judge the truthfulness of the saliency maps. We suggest a protocol to check the validity of the importance attributed to the input and show that the saliency maps obtained sometimes capture the real use of the input features by the network, and sometimes do not. We use this example to discuss how careful we need to be when accepting them as explanation.