Dual-Modality Anchor-Guided Filtering for Test-time Prompt Tuning
This work addresses the view selection bottleneck in TPT for vision-language models, improving robustness under distribution shift.
Test-Time Prompt Tuning (TPT) struggles with selecting beneficial augmented views due to miscalibrated entropy-based filtering under distribution shift. The proposed dual-modality anchor-guided framework uses text and image anchors to filter views based on semantic alignment and confidence, achieving new state-of-the-art performance on 15 benchmark datasets.
Test-Time Prompt Tuning (TPT) adapts vision-language models using augmented views, but its effectiveness is hindered by the challenge of determining which views are beneficial. Standard entropy-based filtering relies on the internal confidence scores of the model, which are often miscalibrated under distribution shift, assigning high confidence to irrelevant crops or background regions while ignoring semantic content. To address this, we propose a dual-modality anchor-guided framework that grounds view selection in semantic evidence. We introduce a text anchor from attribute-rich descriptions, to provide fine-grained class semantics, and an adaptive image anchor that captures evolving test-time statistics. Using these anchors, we filter views based on alignment and confidence, ensuring that only informative views guide adaptation. Moreover, we treat the anchors as auxiliary predictive heads and combine their predictions with the original output in a confidence-weighted ensemble, yielding a stable supervision signal for prompt updates. Extensive experiments on 15 benchmark datasets demonstrate new state-of-the-art performance, highlighting the contribution of anchor-guided supervision as a foundation for robust prompt updates.