TB-AVA: Text as a Semantic Bridge for Audio-Visual Parameter Efficient Finetuning
For audio-visual understanding tasks, TB-AVA provides a parameter-efficient method to leverage text for better cross-modal alignment, achieving SOTA performance.
TB-AVA uses text as a semantic bridge to improve audio-visual alignment, achieving state-of-the-art results on AVE, AVS, and AVVP benchmarks with parameter-efficient fine-tuning.
Audio-visual understanding requires effective alignment between heterogeneous modalities, yet cross-modal correspondence remains challenging when temporally aligned audio and visual signals lack clear semantic correspondence.We propose to use text as a semantic anchor for audio-visual representation learning.To this end, we introduce a parameter-efficient adaptation frameworkbuilt on frozen audio and visual encoders, centered on Text-Bridged Audio-Visual Adapter (TB-AVA), which enables text-mediated interaction between audio and visual streams. At the core of TB-AVA, Gated Semantic Modulation (GSM) selectively modulates feature channels based on text-inferred semantic relevance. We evaluate the proposed approach on multiple benchmarks, including AVE, AVS, and AVVP, where the proposed framework achieves state-of-the-art performance, demonstrating text as an effective semantic anchor for parameter-efficient fine-tuning (PEFT) in audio-visual learning.