Language Model Augmented Relevance Score
This work addresses the need for more reliable evaluation metrics in natural language generation, though it is incremental as it builds on existing methods like BERTScore.
The paper tackled the problem of poor correlation between automated metrics and human judgments in NLG evaluation by proposing MARS, a context-aware metric that uses language models and reinforcement learning to create augmented references, achieving higher correlation with human judgments and better differentiation of adversarial samples compared to seven existing metrics.
Although automated metrics are commonly used to evaluate NLG systems, they often correlate poorly with human judgements. Newer metrics such as BERTScore have addressed many weaknesses in prior metrics such as BLEU and ROUGE, which rely on n-gram matching. These newer methods, however, are still limited in that they do not consider the generation context, so they cannot properly reward generated text that is correct but deviates from the given reference. In this paper, we propose Language Model Augmented Relevance Score (MARS), a new context-aware metric for NLG evaluation. MARS leverages off-the-shelf language models, guided by reinforcement learning, to create augmented references that consider both the generation context and available human references, which are then used as additional references to score generated text. Compared with seven existing metrics in three common NLG tasks, MARS not only achieves higher correlation with human reference judgements, but also differentiates well-formed candidates from adversarial samples to a larger degree.