Historical Test-time Prompt Tuning for Vision Foundation Models
This addresses a robustness issue in test-time adaptation for vision models, which is incremental but important for practical deployment in dynamic environments.
The paper tackles the performance degradation in test-time prompt tuning for vision foundation models when prompts are continuously updated with test data flow, especially under changing domains, and proposes HisTPT, which uses knowledge banks and adaptive retrieval to achieve robust tuning, showing superior performance across tasks like image classification, semantic segmentation, and object detection.
Test-time prompt tuning, which learns prompts online with unlabelled test samples during the inference stage, has demonstrated great potential by learning effective prompts on-the-fly without requiring any task-specific annotations. However, its performance often degrades clearly along the tuning process when the prompts are continuously updated with the test data flow, and the degradation becomes more severe when the domain of test samples changes continuously. We propose HisTPT, a Historical Test-time Prompt Tuning technique that memorizes the useful knowledge of the learnt test samples and enables robust test-time prompt tuning with the memorized knowledge. HisTPT introduces three types of knowledge banks, namely, local knowledge bank, hard-sample knowledge bank, and global knowledge bank, each of which works with different mechanisms for effective knowledge memorization and test-time prompt optimization. In addition, HisTPT features an adaptive knowledge retrieval mechanism that regularizes the prediction of each test sample by adaptively retrieving the memorized knowledge. Extensive experiments show that HisTPT achieves superior prompt tuning performance consistently while handling different visual recognition tasks (e.g., image classification, semantic segmentation, and object detection) and test samples from continuously changing domains.