Beyond Classification: Dynamic Adapter Routing for Continual Multimodal Retrieval
This work tackles the problem of continually updating multimodal retrieval models, which is important for maintaining up-to-date and effective retrieval systems in dynamic environments.
This paper addresses the underexplored problem of continually updating vision-language models for retrieval tasks. It introduces a new evaluation framework for continual multimodal retrieval (CMR) and proposes Dynamic Adapter Routing (DAR), a novel approach that uses prototype-based routing and model merging to achieve superior performance over existing baselines.
While retrieval is a core function of vision-language models, continually updating these models for retrieval tasks remains critically underexplored. Existing work often approaches continual retrieval through the lens of class-incremental learning (CIL), evaluating both standard CIL methods and retrieval-oriented adaptations in settings that may not fully capture the retrieval-specific dynamics. To address this, we introduce a new, principled evaluation framework for continual multimodal retrieval (CMR) spanning diverse visual domains, and systematically evaluate common approaches within this setting. Our empirical analysis shows that standard CIL methods fail to yield meaningful gains in our more challenging scenario. Therefore, we propose Dynamic Adapter Routing (DAR), a novel approach based on adapters selected through prototype-based routing and combined via model merging.DAR achieves superior performance over the previous baselines and demonstrates strong generalization under out-of-distribution evaluation. Our results highlights the unique challenges of CMR and encourages further research in this direction.