ELEGANCE: Efficient LLM Guidance for Audio-Visual Target Speech Extraction
This work addresses the limitation of relying solely on visual cues in audio-visual target speaker extraction, offering a novel integration of linguistic guidance for enhanced performance in complex audio environments.
The paper tackles the problem of audio-visual target speaker extraction by incorporating linguistic knowledge from large language models into existing models, resulting in significant improvements in challenging scenarios such as visual cue impairment and unseen languages.
Audio-visual target speaker extraction (AV-TSE) models primarily rely on visual cues from the target speaker. However, humans also leverage linguistic knowledge, such as syntactic constraints, next word prediction, and prior knowledge of conversation, to extract target speech. Inspired by this observation, we propose ELEGANCE, a novel framework that incorporates linguistic knowledge from large language models (LLMs) into AV-TSE models through three distinct guidance strategies: output linguistic constraints, intermediate linguistic prediction, and input linguistic prior. Comprehensive experiments with RoBERTa, Qwen3-0.6B, and Qwen3-4B on two AV-TSE backbones demonstrate the effectiveness of our approach. Significant improvements are observed in challenging scenarios, including visual cue impaired, unseen languages, target speaker switches, increased interfering speakers, and out-of-domain test set. Demo page: https://alexwxwu.github.io/ELEGANCE/.