Multiple Consistency-guided Test-Time Adaptation for Contrastive Audio-Language Models with Unlabeled Audio
This work addresses the challenge of improving domain performance for audio-language models without annotations, which is incremental as it builds on existing test-time adaptation methods.
The paper tackles the problem of test-time adaptation for audio-language models in zero-shot classification, where previous methods often get stuck in incorrect predictions, and proposes a method using multiple consistency guidance on prompt learning without annotated labels, resulting in an average zero-shot performance improvement of 4.41% (up to 7.50%) across 12 downstream tasks compared to state-of-the-art models.
One fascinating aspect of pre-trained Audio-Language Models (ALMs) learning is their impressive zero-shot generalization capability and test-time adaptation (TTA) methods aiming to improve domain performance without annotations. However, previous test time adaptation (TTA) methods for ALMs in zero-shot classification tend to be stuck in incorrect model predictions. In order to further boost the performance, we propose multiple guidance on prompt learning without annotated labels. First, guidance of consistency on both context tokens and domain tokens of ALMs is set. Second, guidance of both consistency across multiple augmented views of each single test sample and contrastive learning across different test samples is set. Third, we propose a corresponding end-end learning framework for the proposed test-time adaptation method without annotated labels. We extensively evaluate our approach on 12 downstream tasks across domains, our proposed adaptation method leads to 4.41% (max 7.50%) average zero-shot performance improvement in comparison with the state-of-the-art models.