BMX: Boosting Natural Language Generation Metrics with Explainability
This work addresses the need for more accurate evaluation metrics in natural language generation, particularly for summarization, though it is incremental as it builds on existing explainability methods.
The paper tackles the problem of improving natural language generation evaluation metrics by leveraging explainability, specifically converting feature importance explanations into segment-level scores and combining them with original metrics, resulting in an average improvement of 0.087 points in Spearman correlation on SummEval for summarization.
State-of-the-art natural language generation evaluation metrics are based on black-box language models. Hence, recent works consider their explainability with the goals of better understandability for humans and better metric analysis, including failure cases. In contrast, our proposed method BMX: Boosting Natural Language Generation Metrics with explainability explicitly leverages explanations to boost the metrics' performance. In particular, we perceive feature importance explanations as word-level scores, which we convert, via power means, into a segment-level score. We then combine this segment-level score with the original metric to obtain a better metric. Our tests show improvements for multiple metrics across MT and summarization datasets. While improvements in machine translation are small, they are strong for summarization. Notably, BMX with the LIME explainer and preselected parameters achieves an average improvement of 0.087 points in Spearman correlation on the system-level evaluation of SummEval.