Incorporating Linguistic Constraints from External Knowledge Source for Audio-Visual Target Speech Extraction
This work addresses audio-visual speech extraction for applications like hearing aids or communication systems, offering an incremental improvement by integrating external linguistic knowledge.
The paper tackled the problem of audio-visual target speaker extraction by incorporating linguistic constraints from pre-trained speech-language or language models as additional supervision, resulting in improved speech quality and intelligibility without extra inference cost, with robust gains in multi-language and visual-impaired scenarios.
Audio-visual target speaker extraction (AV-TSE) models primarily rely on target visual cues to isolate the target speaker's voice from others. We know that humans leverage linguistic knowledge, such as syntax and semantics, to support speech perception. Inspired by this, we explore the potential of pre-trained speech-language models (PSLMs) and pre-trained language models (PLMs) as auxiliary knowledge sources for AV-TSE. In this study, we propose incorporating the linguistic constraints from PSLMs or PLMs for the AV-TSE model as additional supervision signals. Without introducing any extra computational cost during inference, the proposed approach consistently improves speech quality and intelligibility. Furthermore, we evaluate our method in multi-language settings and visual cue-impaired scenarios and show robust performance gains.