Scope: Selective Cross-modal Orchestration of Visual Perception Experts
This addresses the problem of high inference costs in multi-encoder vision-language models for AI researchers and practitioners, offering a novel approach that is not incremental but challenges the prevailing paradigm.
The paper tackles the inefficiency of stacking multiple vision encoders in vision-language models by proposing SCOPE, a framework that dynamically selects one specialized encoder per image-text pair, reducing compute by 24-49% while outperforming models using all encoders simultaneously.
Vision-language models (VLMs) benefit from multiple vision encoders, but naively stacking them yields diminishing returns while multiplying inference costs. We propose SCOPE, a Mixture-of-Encoders (MoEnc) framework that dynamically selects one specialized encoder per image-text pair via instance-level routing, unlike token-level routing in traditional MoE. SCOPE maintains a shared encoder and a pool of routed encoders. A lightweight router uses cross-attention between text prompts and shared visual features to select the optimal encoder from the routed encoders. To train this router, we introduce dual entropy regularization with auxiliary losses to balance dataset-level load distribution with instance-level routing confidence. Remarkably, SCOPE with one shared plus one routed encoder outperforms models using all four extra encoders simultaneously, while reducing compute by 24-49\%. This demonstrates that intelligent encoder selection beats brute-force aggregation, challenging the prevailing paradigm in multi-encoder VLMs.