Do Audio-Language Models Understand Linguistic Variations?
This addresses a robustness issue in audio-text retrieval for users relying on natural language queries, though it is incremental as it builds on existing CLAP architectures.
The paper tackles the problem that existing audio-language models struggle with linguistic variations in textual queries, and proposes RobustCLAP, which improves text-to-audio retrieval performance by 0.8%-13% across benchmarks.
Open-vocabulary audio language models (ALMs), like Contrastive Language Audio Pretraining (CLAP), represent a promising new paradigm for audio-text retrieval using natural language queries. In this paper, for the first time, we perform controlled experiments on various benchmarks to show that existing ALMs struggle to generalize to linguistic variations in textual queries. To address this issue, we propose RobustCLAP, a novel and compute-efficient technique to learn audio-language representations agnostic to linguistic variations. Specifically, we reformulate the contrastive loss used in CLAP architectures by introducing a multi-view contrastive learning objective, where paraphrases are treated as different views of the same audio scene and use this for training. Our proposed approach improves the text-to-audio retrieval performance of CLAP by 0.8%-13% across benchmarks and enhances robustness to linguistic variation.