SingMOS-Pro: An Comprehensive Benchmark for Singing Quality Assessment
This provides a standardized benchmark for researchers in singing voice generation to objectively assess quality, though it is incremental as it builds on a previous version.
The authors tackled the challenge of evaluating singing quality in voice generation by introducing SingMOS-Pro, a comprehensive dataset with 7,981 singing clips from 41 models, annotated for lyrics, melody, and overall quality by professionals, and established baselines for future research.
Singing voice generation progresses rapidly, yet evaluating singing quality remains a critical challenge. Human subjective assessment, typically in the form of listening tests, is costly and time consuming, while existing objective metrics capture only limited perceptual aspects. In this work, we introduce SingMOS-Pro, a dataset for automatic singing quality assessment. Building on our preview version SingMOS, which provides only overall ratings, SingMOS-Pro expands annotations of the additional part to include lyrics, melody, and overall quality, offering broader coverage and greater diversity. The dataset contains 7,981 singing clips generated by 41 models across 12 datasets, spanning from early systems to recent advances. Each clip receives at least five ratings from professional annotators, ensuring reliability and consistency. Furthermore, we explore how to effectively utilize MOS data annotated under different standards and benchmark several widely used evaluation methods from related tasks on SingMOS-Pro, establishing strong baselines and practical references for future research. The dataset can be accessed at https://huggingface.co/datasets/TangRain/SingMOS-Pro.