Improving Metrics for Speech Translation
This addresses metric reliability for speech translation systems, particularly for low-resource languages like Swiss German, but is incremental as it builds on existing metric frameworks.
The paper tackles the problem of misleading speech translation metrics like WER, CER, and BLEU when only a single reference is available, by introducing Parallel Paraphrasing (Para_both) as an augmentation method, and shows it significantly improves correlation with human quality perception on new Swiss German speech-to-text datasets.
We introduce Parallel Paraphrasing ($\text{Para}_\text{both}$), an augmentation method for translation metrics making use of automatic paraphrasing of both the reference and hypothesis. This method counteracts the typically misleading results of speech translation metrics such as WER, CER, and BLEU if only a single reference is available. We introduce two new datasets explicitly created to measure the quality of metrics intended to be applied to Swiss German speech-to-text systems. Based on these datasets, we show that we are able to significantly improve the correlation with human quality perception if our method is applied to commonly used metrics.