PRISM: Synergizing Vision Foundation Models via Self-organized Expert Specialization
For practitioners needing to combine multiple vision models efficiently, PRISM offers a scalable method to integrate diverse visual knowledge without performance degradation.
PRISM introduces a dual-stream Mixture-of-Experts framework to synergize diverse Vision Foundation Models, achieving new state-of-the-art results on PASCAL-Context and NYUD-v2 by mitigating negative transfer through modular specialization.
Unifying the complementary strengths of diverse Vision Foundation Models (VFMs) into a single efficient model is highly desirable but challenged by the negative transfer inherent in monolithic distillation. To address these feature conflicts, we introduce \textbf{PRISM}, a novel dual-stream Mixture-of-Experts (MoE) framework that synergizes VFMs via modular specialization. We propose a two-stage paradigm: (1) expertise deconstruction, where a teacher-conditional router guides experts to specialize in distinct representational subspaces to mitigate interference, followed by (2) dynamic recomposition, where the router learns to assemble these experts into tailored computational pathways for downstream tasks. Experiments on PASCAL-Context and NYUD-v2 show that \textbf{PRISM} establishes a new state of the art, validating that sparse, emergent specialization is a scalable approach for integrating diverse visual knowledge.