Comparison and Adaptation of Automatic Evaluation Metrics for Quality Assessment of Re-Speaking
This work addresses the problem of automating quality assessment for re-speaking, which is currently done manually, but it is incremental as it applies existing metrics from machine translation to a new domain.
The paper compared automatic evaluation metrics (BLEU, EBLEU, NIST, METEOR, METEOR-PL, TER, RIBES) for assessing the quality of re-speaking, a method for generating live subtitles, by matching them to a human-derived NER metric.
Re-speaking is a mechanism for obtaining high quality subtitles for use in live broadcast and other public events. Because it relies on humans performing the actual re-speaking, the task of estimating the quality of the results is non-trivial. Most organisations rely on humans to perform the actual quality assessment, but purely automatic methods have been developed for other similar problems, like Machine Translation. This paper will try to compare several of these methods: BLEU, EBLEU, NIST, METEOR, METEOR-PL, TER and RIBES. These will then be matched to the human-derived NER metric, commonly used in re-speaking.