BERTScore: Evaluating Text Generation with BERT
This addresses the need for more reliable automatic evaluation metrics in natural language processing, particularly for researchers and practitioners in machine translation and image captioning, though it is incremental as it builds on existing metric concepts with contextual embeddings.
The authors tackled the problem of evaluating text generation by proposing BERTScore, a metric that uses contextual embeddings for token similarity instead of exact matches. The result showed that BERTScore correlates better with human judgments and provides stronger model selection performance than existing metrics, as evaluated on 363 machine translation and image captioning systems.
We propose BERTScore, an automatic evaluation metric for text generation. Analogously to common metrics, BERTScore computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTScore correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BERTScore is more robust to challenging examples when compared to existing metrics.