PrefGen: Multimodal Preference Learning for Preference-Conditioned Image Generation
This work addresses the challenge of personalized image generation for users, offering a novel method to encode nuanced preferences, though it is incremental in building on existing diffusion and MLLM techniques.
The paper tackled the problem of adapting generative models to individual user preferences for image generation, proposing a multimodal framework that uses MLLMs to extract and inject user representations into diffusion models, resulting in substantial improvements in image quality and preference alignment over baselines.
Preference-conditioned image generation seeks to adapt generative models to individual users, producing outputs that reflect personal aesthetic choices beyond the given textual prompt. Despite recent progress, existing approaches either fail to capture nuanced user preferences or lack effective mechanisms to encode personalized visual signals. In this work, we propose a multimodal framework that leverages multimodal large language models (MLLMs) to extract rich user representations and inject them into diffusion-based image generation. We train the MLLM with a preference-oriented visual question answering task to capture fine-grained semantic cues. To isolate preference-relevant features, we introduce two complementary probing tasks: inter-user discrimination to distinguish between different users, and intra-user discrimination to separate liked from disliked content. To ensure compatibility with diffusion text encoders, we design a maximum mean discrepancy-based alignment loss that bridges the modality gap while preserving multimodal structure. The resulting embeddings are used to condition the generator, enabling faithful adherence to both prompts and user preferences. Extensive experiments demonstrate that our method substantially outperforms strong baselines in both image quality and preference alignment, highlighting the effectiveness of representation extraction and alignment for personalized generation.