Chinese-LiPS: A Chinese audio-visual speech recognition dataset with Lip-reading and Presentation Slides
This addresses the need for richer multimodal datasets in Chinese AVSR, though it is incremental as it extends existing AVSR approaches with additional visual cues.
The authors tackled the problem of limited visual cues in audio-visual speech recognition by creating Chinese-LiPS, a dataset with 100 hours of speech, video, and transcription that includes both lip-reading and presentation slides as visual modalities, and their LiPS-AVSR pipeline achieved combined performance improvements of about 35% over audio-only ASR.
Incorporating visual modalities to assist Automatic Speech Recognition (ASR) tasks has led to significant improvements. However, existing Audio-Visual Speech Recognition (AVSR) datasets and methods typically rely solely on lip-reading information or speaking contextual video, neglecting the potential of combining these different valuable visual cues within the speaking context. In this paper, we release a multimodal Chinese AVSR dataset, Chinese-LiPS, comprising 100 hours of speech, video, and corresponding manual transcription, with the visual modality encompassing both lip-reading information and the presentation slides used by the speaker. Based on Chinese-LiPS, we develop a simple yet effective pipeline, LiPS-AVSR, which leverages both lip-reading and presentation slide information as visual modalities for AVSR tasks. Experiments show that lip-reading and presentation slide information improve ASR performance by approximately 8\% and 25\%, respectively, with a combined performance improvement of about 35\%. The dataset is available at https://kiri0824.github.io/Chinese-LiPS/