Test-Time Adaptation of Vision-Language Models for Open-Vocabulary Semantic Segmentation
This work addresses a domain-specific problem for researchers and practitioners in computer vision by providing a plug-and-play solution for adapting vision-language models to segmentation tasks without additional data, though it is incremental as it extends existing TTA concepts to a new task.
The paper tackles the problem of test-time adaptation for open-vocabulary semantic segmentation, which was previously overlooked, by proposing a novel method that integrates multi-level features and multi-prompt entropy minimization, achieving significant gains over classification baselines across 87 distinct test scenarios.
Recently, test-time adaptation has attracted wide interest in the context of vision-language models for image classification. However, to the best of our knowledge, the problem is completely overlooked in dense prediction tasks such as Open-Vocabulary Semantic Segmentation (OVSS). In response, we propose a novel TTA method tailored to adapting VLMs for segmentation during test time. Unlike TTA methods for image classification, our Multi-Level and Multi-Prompt (MLMP) entropy minimization integrates features from intermediate vision-encoder layers and is performed with different text-prompt templates at both the global CLS token and local pixel-wise levels. Our approach could be used as plug-and-play for any segmentation network, does not require additional training data or labels, and remains effective even with a single test sample. Furthermore, we introduce a comprehensive OVSS TTA benchmark suite, which integrates a rigorous evaluation protocol, nine segmentation datasets, 15 common synthetic corruptions, and additional real and rendered domain shifts, \textbf{with a total of 87 distinct test scenarios}, establishing a standardized and comprehensive testbed for future TTA research in open-vocabulary segmentation. Our experiments on this suite demonstrate that our segmentation-tailored method consistently delivers significant gains over direct adoption of TTA classification baselines. Code and data are available at https://github.com/dosowiechi/MLMP.