OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
This addresses a practical challenge in adapting models to real-world scenarios without prior domain knowledge, though it is incremental in improving evaluation and optimization methods for existing prompt tuning paradigms.
The paper tackles the problem of evaluating and optimizing open-world prompt tuning for Vision-Language Models, where current metrics fail to unify detection and classification tasks under varying domain ratios, and proposes OpenworldAUC as a new metric and Gated Mixture-of-Prompts (GMoP) as a method, achieving state-of-the-art performance on 15 benchmarks.
Prompt tuning adapts Vision-Language Models like CLIP to open-world tasks with minimal training costs. In this direction, one typical paradigm evaluates model performance separately on known classes (i.e., base domain) and unseen classes (i.e., new domain). However, real-world scenarios require models to handle inputs without prior domain knowledge. This practical challenge has spurred the development of open-world prompt tuning, which demands a unified evaluation of two stages: 1) detecting whether an input belongs to the base or new domain (P1), and 2) classifying the sample into its correct class (P2). What's more, as domain distributions are generally unknown, a proper metric should be insensitive to varying base/new sample ratios (P3). However, we find that current metrics, including HM, overall accuracy, and AUROC, fail to satisfy these three properties simultaneously. To bridge this gap, we propose OpenworldAUC, a unified metric that jointly assesses detection and classification through pairwise instance comparisons. To optimize OpenworldAUC effectively, we introduce Gated Mixture-of-Prompts (GMoP), which employs domain-specific prompts and a gating mechanism to dynamically balance detection and classification. Theoretical guarantees ensure generalization of GMoP under practical conditions. Experiments on 15 benchmarks in open-world scenarios show GMoP achieves SOTA performance on OpenworldAUC and other metrics. We release the code at https://github.com/huacong/OpenworldAUC