Mod-Adapter: Tuning-Free and Versatile Multi-concept Personalization via Modulation Adapter
This work addresses the challenge of efficiently personalizing text-to-image generation for multiple concepts, including abstract ones, without time-consuming fine-tuning, which is significant for users in creative and AI-driven content creation domains.
The paper tackles the problem of multi-concept personalization in text-to-image generation, which struggles with abstract concepts and requires test-time fine-tuning, by proposing a tuning-free method called Mod-Adapter that leverages modulation in Diffusion Transformers to customize both object and abstract concepts, achieving state-of-the-art performance as validated by quantitative, qualitative, and human evaluations.
Personalized text-to-image generation aims to synthesize images of user-provided concepts in diverse contexts. Despite recent progress in multi-concept personalization, most are limited to object concepts and struggle to customize abstract concepts (e.g., pose, lighting). Some methods have begun exploring multi-concept personalization supporting abstract concepts, but they require test-time fine-tuning for each new concept, which is time-consuming and prone to overfitting on limited training images. In this work, we propose a novel tuning-free method for multi-concept personalization that can effectively customize both object and abstract concepts without test-time fine-tuning. Our method builds upon the modulation mechanism in pre-trained Diffusion Transformers (DiTs) model, leveraging the localized and semantically meaningful properties of the modulation space. Specifically, we propose a novel module, Mod-Adapter, to predict concept-specific modulation direction for the modulation process of concept-related text tokens. It introduces vision-language cross-attention for extracting concept visual features, and Mixture-of-Experts (MoE) layers that adaptively map the concept features into the modulation space. Furthermore, to mitigate the training difficulty caused by the large gap between the concept image space and the modulation space, we introduce a VLM-guided pre-training strategy that leverages the strong image understanding capabilities of vision-language models to provide semantic supervision signals. For a comprehensive comparison, we extend a standard benchmark by incorporating abstract concepts. Our method achieves state-of-the-art performance in multi-concept personalization, supported by quantitative, qualitative, and human evaluations.