The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech
This work addresses the need for accurate MOS prediction in synthetic speech evaluation, which is crucial for speech synthesis research and applications, representing an incremental improvement through feature fusion.
The paper tackled the problem of predicting naturalness mean opinion scores (MOS) for high-quality synthetic speech by developing the T05 system, which achieved first place in 7 out of 16 evaluation metrics and second place in the remaining 9 metrics in the VoiceMOS Challenge 2024 Track 1.
We present our system (denoted as T05) for the VoiceMOS Challenge (VMC) 2024. Our system was designed for the VMC 2024 Track 1, which focused on the accurate prediction of naturalness mean opinion score (MOS) for high-quality synthetic speech. In addition to a pretrained self-supervised learning (SSL)-based speech feature extractor, our system incorporates a pretrained image feature extractor to capture the difference of synthetic speech observed in speech spectrograms. We first separately train two MOS predictors that use either of an SSL-based or spectrogram-based feature. Then, we fine-tune the two predictors for better MOS prediction using the fusion of two extracted features. In the VMC 2024 Track 1, our T05 system achieved first place in 7 out of 16 evaluation metrics and second place in the remaining 9 metrics, with a significant difference compared to those ranked third and below. We also report the results of our ablation study to investigate essential factors of our system.