DynaPrompt: Dynamic Test-Time Prompt Tuning
This work addresses a specific bottleneck in zero-shot generalization for vision-language models, offering an incremental improvement over existing test-time tuning methods.
The paper tackles the problem of error accumulation in test-time prompt tuning for vision-language models by proposing DynaPrompt, which dynamically selects and optimizes prompts using a buffer and selection metrics, achieving improved performance across fourteen datasets.
Test-time prompt tuning enhances zero-shot generalization of vision-language models but tends to ignore the relatedness among test samples during inference. Online test-time prompt tuning provides a simple way to leverage the information in previous test samples, albeit with the risk of prompt collapse due to error accumulation. To enhance test-time prompt tuning, we propose DynaPrompt, short for dynamic test-time prompt tuning, exploiting relevant data distribution information while reducing error accumulation. Built on an online prompt buffer, DynaPrompt adaptively selects and optimizes the relevant prompts for each test sample during tuning. Specifically, we introduce a dynamic prompt selection strategy based on two metrics: prediction entropy and probability difference. For unseen test data information, we develop dynamic prompt appending, which allows the buffer to append new prompts and delete the inactive ones. By doing so, the prompts are optimized to exploit beneficial information on specific test data, while alleviating error accumulation. Experiments on fourteen datasets demonstrate the effectiveness of dynamic test-time prompt tuning.