Perception Score, A Learned Metric for Open-ended Text Generation Evaluation
This addresses the challenge of low correlation with human judgment in existing metrics like BLEU for researchers and practitioners in natural language generation.
The paper tackles the problem of automatic evaluation for open-ended text generation by proposing Perception Score, a learned metric that measures overall quality holistically and includes uncertainty estimation, achieving state-of-the-art results on four generation tasks.
Automatic evaluation for open-ended natural language generation tasks remains a challenge. Existing metrics such as BLEU show a low correlation with human judgment. We propose a novel and powerful learning-based evaluation metric: Perception Score. The method measures the overall quality of the generation and scores holistically instead of only focusing on one evaluation criteria, such as word overlapping. Moreover, it also shows the amount of uncertainty about its evaluation result. By connecting the uncertainty, Perception Score gives a more accurate evaluation for the generation system. Perception Score provides state-of-the-art results on two conditional generation tasks and two unconditional generation tasks.