Generalizable Prompt Tuning for Audio-Language Models via Semantic Expansion
Addresses the generalization problem in prompt tuning for audio-language models, which is a known bottleneck for adapting VLMs/ALMs to new tasks.
Prompt tuning for audio-language models suffers from a base-new tradeoff due to disrupted semantic structure. The proposed SEPT framework regularizes the embedding space using semantic neighbors from LLMs, achieving consistent generalization improvements across multiple baselines.
Prompt tuning has achieved remarkable progress in vision-language models (VLMs) and is recently being adopted for audio-language models (ALMs). However, its generalization ability in ALMs remains largely underexplored. We observe that conventional prompt tuning for ALMs also suffers from the Base-New Tradeoff, and we identify that this issue stems from the disrupted semantic structure of the embedding space. To address this issue, we propose Semantically Expanded Prompt Tuning (SEPT)-a plug-and-play framework that explicitly regularizes the prompt embedding space by incorporating semantic neighbors generated by large language models. SEPT introduces a novel semantic expansion loss with margin constraints that promote intra-class compactness and inter-class separability, thereby enhancing the semantic structure of the prompt embedding space. For comprehensive evaluation, we establish the first benchmark setup for prompt generalization in ALMs, covering both base-to-new generalization and cross-dataset transferability. Extensive experiments demonstrate that SEPT consistently improves generalization performance across multiple prompt tuning baselines, while maintaining computational cost during inference.