Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-Visual Language Models
For developers of multimodal AI systems, this work addresses a critical reliability bottleneck in audio-visual understanding by mitigating cross-modal hallucination, though the improvement is incremental as it builds on preference optimization techniques.
Audio-Visual Language Models suffer from video-driven audio hallucination, where visual shortcuts override true auditory evidence. The proposed Audio-Contrastive Preference Optimization (ACPO) reduces audio hallucination by penalizing visual descriptions masquerading as audio facts and generation invariant to the true audio signal, achieving more faithful audio grounding without harming overall multimodal performance.
While Audio-Visual Language Models (AVLMs) have achieved remarkable progress over recent years, their reliability is bottlenecked by cross-modal hallucination. A particularly pervasive manifestation is video-driven audio hallucination: models routinely exploit visual shortcuts to hallucinate expected sounds, discarding true auditory evidence. To counteract this deeply ingrained visual dominance, we propose Audio-Contrastive Preference Optimization (ACPO). This dual-axis preference learning framework introduces an output-contrastive objective to penalize visual descriptions masquerading as audio facts, alongside an input-contrastive objective that swaps audio tracks to explicitly penalize generation invariant to the true auditory signal. Extensive experiments demonstrate that ACPO establishes highly faithful audio grounding and mitigates audio hallucination without compromising overarching multimodal capabilities.