Scaling Audio-Visual Quality Assessment Dataset via Crowdsourcing
This work is significant for researchers in multimodal perception and quality assessment, as it provides a larger and more diverse dataset to overcome the limitations of previous small-scale datasets.
The paper addresses the limitations of existing Audio-Visual Quality Assessment (AVQA) datasets by proposing a crowdsourcing framework for dataset construction. This approach led to the creation of YT-NTU-AVQ, the largest and most diverse AVQA dataset to date, comprising 1,620 user-generated audio and video sequences.
Audio-visual quality assessment (AVQA) research has been stalled by limitations of existing datasets: they are typically small in scale, with insufficient diversity in content and quality, and annotated only with overall scores. These shortcomings provide limited support for model development and multimodal perception research. We propose a practical approach for AVQA dataset construction. First, we design a crowdsourced subjective experiment framework for AVQA, breaks the constraints of in-lab settings and achieves reliable annotation across varied environments. Second, a systematic data preparation strategy is further employed to ensure broad coverage of both quality levels and semantic scenarios. Third, we extend the dataset with additional annotations, enabling research on multimodal perception mechanisms and their relation to content. Finally, we validate this approach through YT-NTU-AVQ, the largest and most diverse AVQA dataset to date, consisting of 1,620 user-generated audio and video (A/V) sequences. The dataset and platform code are available at https://github.com/renyu12/YT-NTU-AVQ