A Multilingual Perspective Towards the Evaluation of Attribution Methods in Natural Language Inference
This work addresses the lack of multilingual evaluation in explainable NLP, providing a dataset and methods for researchers, but it is incremental as it builds on existing attribution techniques.
The paper tackled the problem of evaluating attribution methods for Natural Language Inference (NLI) by introducing a multilingual approach, showing that the best methods for faithfulness and plausibility differ.
Most evaluations of attribution methods focus on the English language. In this work, we present a multilingual approach for evaluating attribution methods for the Natural Language Inference (NLI) task in terms of faithfulness and plausibility. First, we introduce a novel cross-lingual strategy to measure faithfulness based on word alignments, which eliminates the drawbacks of erasure-based evaluations.We then perform a comprehensive evaluation of attribution methods, considering different output mechanisms and aggregation methods. Finally, we augment the XNLI dataset with highlight-based explanations, providing a multilingual NLI dataset with highlights, to support future exNLP studies. Our results show that attribution methods performing best for plausibility and faithfulness are different.