TSPE: Task-Specific Prompt Ensemble for Improved Zero-Shot Audio Classification
This work addresses the challenge of enhancing audio-language models for zero-shot classification in audio domains, representing an incremental advancement in prompt engineering.
The paper tackles the problem of improving zero-shot audio classification by introducing TSPE, a training-free prompting method that customizes prompts for diverse tasks, resulting in absolute performance improvements of 1.23-16.36% across 12 datasets.
Audio-language models (ALMs) excel in zero-shot audio classification, a task where models classify previously unseen audio clips at test time by leveraging descriptive natural language prompts. We introduce TSPE (Task-Specific Prompt Ensemble), a simple, training-free hard prompting method that boosts ALEs' zero-shot performance by customizing prompts for diverse audio classification tasks. Rather than using generic template-based prompts like "Sound of a car" we generate context-rich prompts, such as "Sound of a car coming from a tunnel". Specifically, we leverage label information to identify suitable sound attributes, such as "loud" and "feeble", and appropriate sound sources, such as "tunnel" and "street" and incorporate this information into the prompts used by Audio-Language Models (ALMs) for audio classification. Further, to enhance audio-text alignment, we perform prompt ensemble across TSPE-generated task-specific prompts. When evaluated on 12 diverse audio classification datasets, TSPE improves performance across ALMs by showing an absolute improvement of 1.23-16.36% over vanilla zero-shot evaluation.