DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answering
This addresses the problem of over-fitting and lack of interpretability in NLG evaluation metrics for researchers and practitioners, though it is incremental as it builds on existing instruction-tuned models.
The authors tackled the challenges of generalization and interpretability in natural language generation evaluation by proposing DecompEval, a metric that formulates evaluation as an unsupervised decomposed question answering task using instruction-tuned language models, achieving state-of-the-art performance in untrained metrics for text summarization and dialogue generation.
Existing evaluation metrics for natural language generation (NLG) tasks face the challenges on generalization ability and interpretability. Specifically, most of the well-performed metrics are required to train on evaluation datasets of specific NLG tasks and evaluation dimensions, which may cause over-fitting to task-specific datasets. Furthermore, existing metrics only provide an evaluation score for each dimension without revealing the evidence to interpret how this score is obtained. To deal with these challenges, we propose a simple yet effective metric called DecompEval. This metric formulates NLG evaluation as an instruction-style question answering task and utilizes instruction-tuned pre-trained language models (PLMs) without training on evaluation datasets, aiming to enhance the generalization ability. To make the evaluation process more interpretable, we decompose our devised instruction-style question about the quality of generated texts into the subquestions that measure the quality of each sentence. The subquestions with their answers generated by PLMs are then recomposed as evidence to obtain the evaluation result. Experimental results show that DecompEval achieves state-of-the-art performance in untrained metrics for evaluating text summarization and dialogue generation, which also exhibits strong dimension-level / task-level generalization ability and interpretability.