SubER: A Metric for Automatic Evaluation of Subtitle Quality
This addresses the need for a comprehensive evaluation metric for subtitle quality, which is incremental as it builds on existing methods by integrating multiple features.
The paper tackles the problem of evaluating automatically generated subtitles by proposing SubER, a metric based on edit distance with shifts that accounts for transcription, translation, segmentation, and timing quality. It shows high correlation with human assessment and post-editing effort, outperforming baseline metrics like WER and BLEU.
This paper addresses the problem of evaluating the quality of automatically generated subtitles, which includes not only the quality of the machine-transcribed or translated speech, but also the quality of line segmentation and subtitle timing. We propose SubER - a single novel metric based on edit distance with shifts that takes all of these subtitle properties into account. We compare it to existing metrics for evaluating transcription, translation, and subtitle quality. A careful human evaluation in a post-editing scenario shows that the new metric has a high correlation with the post-editing effort and direct human assessment scores, outperforming baseline metrics considering only the subtitle text, such as WER and BLEU, and existing methods to integrate segmentation and timing features.