T5Score: Discriminative Fine-tuning of Generative Evaluation Metrics
This work addresses the need for more accurate and versatile evaluation metrics in natural language processing, particularly for generated text, by integrating supervised and unsupervised training signals, though it is incremental in building upon existing paradigms.
The paper tackles the problem of evaluating generated text by combining discriminative and generative approaches into a unified metric called T5Score, which achieves the best performance on all tested datasets against existing top-scoring metrics at the segment level across 5 datasets, 19 languages, and 280 systems.
Modern embedding-based metrics for evaluation of generated text generally fall into one of two paradigms: discriminative metrics that are trained to directly predict which outputs are of higher quality according to supervised human annotations, and generative metrics that are trained to evaluate text based on the probabilities of a generative model. Both have their advantages; discriminative metrics are able to directly optimize for the problem of distinguishing between good and bad outputs, while generative metrics can be trained using abundant raw text. In this paper, we present a framework that combines the best of both worlds, using both supervised and unsupervised signals from whatever data we have available. We operationalize this idea by training T5Score, a metric that uses these training signals with mT5 as the backbone. We perform an extensive empirical comparison with other existing metrics on 5 datasets, 19 languages and 280 systems, demonstrating the utility of our method. Experimental results show that: T5Score achieves the best performance on all datasets against existing top-scoring metrics at the segment level. We release our code and models at https://github.com/qinyiwei/T5Score.