Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing
This work addresses the need for systematic organization and improvement of datasets to advance explainable NLP research, but it is incremental as it reviews existing data rather than introducing new methods.
The paper reviews 65 datasets for explainable NLP, categorizing them into three types of textual explanations (highlights, free-text, structured), analyzes annotation methodologies, and provides recommendations for future dataset collection.
Explainable NLP (ExNLP) has increasingly focused on collecting human-annotated textual explanations. These explanations are used downstream in three ways: as data augmentation to improve performance on a predictive task, as supervision to train models to produce explanations for their predictions, and as a ground-truth to evaluate model-generated explanations. In this review, we identify 65 datasets with three predominant classes of textual explanations (highlights, free-text, and structured), organize the literature on annotating each type, identify strengths and shortcomings of existing collection methodologies, and give recommendations for collecting ExNLP datasets in the future.