Gate-and-Merge: Zero-shot Compositional Personalization of Vision Language Models
This work addresses the need for compositional personalization in VLMs without co-occurrence training, offering a practical zero-shot solution for users who want to combine multiple personalized concepts.
Gate-and-Merge enables zero-shot compositional personalization of VLMs by learning each concept as an independent LoRA adapter and merging them at inference with a gating mechanism, achieving consistent gains across multiple tasks.
This paper tackles compositional personalization of vision-language models (VLMs). In this problem, multiple user-defined concepts must be recognized or described jointly at test time. We introduce Gate-and-Merge, a zero-shot framework that enables compositional personalization without the need for co-occurrence training. During personalization, each concept is learned independently as a lightweight LoRA adapter, paired with a concept token. The base model remains unchanged and concepts are kept disentangled. At inference, we enable composition by merging concept-specific LoRA updates directly in weight space. To suppress irrelevant activations and prevent interference, a gating mechanism is employed to estimate textual and visual cues and select only the modules that contribute to the prediction. We further stabilize composition by combining only the most meaningful and mutually consistent updates, helping preserve each concept's identity. Our quantitative and qualitative analyses show consistent gains in performance across multiple personalization tasks in both single-concept and compositional settings.