Attacking UTMOS: Probing the Robustness of a Speech Quality Assessment Model
For researchers using UTMOS for speech quality assessment, this work reveals failure modes and underscores the need for robustness analysis in DNN-based metrics.
UTMOS, a deep neural network-based speech quality assessment metric, is attacked to probe its robustness. Score-preserving attacks successfully degrade perceived quality while maintaining predicted scores, and quality-preserving attacks in the EnCodec latent space show the best chance of lowering predicted scores without affecting perceived quality.
UTMOS has become one of the most commonly used deep neural network-based speech quality assessment (SQA) metrics in speech processing research. In this paper, we attack UTMOS to probe its robustness. Starting from high-quality speech samples, we optimize the input in two directions: a score-preserving attack, which degrades perceived quality while maintaining the predicted score, and a quality-preserving attack, which lowers the predicted score while maintaining perceived quality. We consider three input spaces: raw waveform, mel spectrogram with a HiFi-GAN vocoder, and the latent space of EnCodec, a neural audio codec. Experimental results show that score-preserving attacks are effective against UTMOS. Although perfect quality-preserving attacks are more difficult, optimization in the EnCodec latent space provides the best chance of success. These results reveal failure modes of UTMOS and highlight the importance of robustness analysis for DNN-based SQA metrics.