MOSPC: MOS Prediction Based on Pairwise Comparison
This work addresses the problem of reducing labor costs in speech quality evaluation for researchers and practitioners, though it is incremental as it builds on existing MOS prediction methods.
The paper tackles the challenge of accurately ranking speech quality when Mean Opinion Score (MOS) values are close by proposing MOSPC, a framework for MOS prediction based on pairwise comparison, which outperforms baselines on correlation metrics and ranking accuracy in experiments on BVCC and VCC2018 datasets.
As a subjective metric to evaluate the quality of synthesized speech, Mean opinion score~(MOS) usually requires multiple annotators to score the same speech. Such an annotation approach requires a lot of manpower and is also time-consuming. MOS prediction model for automatic evaluation can significantly reduce labor cost. In previous works, it is difficult to accurately rank the quality of speech when the MOS scores are close. However, in practical applications, it is more important to correctly rank the quality of synthesis systems or sentences than simply predicting MOS scores. Meanwhile, as each annotator scores multiple audios during annotation, the score is probably a relative value based on the first or the first few speech scores given by the annotator. Motivated by the above two points, we propose a general framework for MOS prediction based on pair comparison (MOSPC), and we utilize C-Mixup algorithm to enhance the generalization performance of MOSPC. The experiments on BVCC and VCC2018 show that our framework outperforms the baselines on most of the correlation coefficient metrics, especially on the metric KTAU related to quality ranking. And our framework also surpasses the strong baseline in ranking accuracy on each fine-grained segment. These results indicate that our framework contributes to improving the ranking accuracy of speech quality.