Premier: Personalized Preference Modulation with Learnable User Embedding in Text-to-Image Generation
This addresses the challenge of personalizing image generation for users, though it is incremental as it builds on existing preference modulation approaches.
The paper tackles the problem of capturing nuanced user preferences in text-to-image generation by introducing Premier, a framework that uses learnable user embeddings and a preference adapter to modulate the generative process, resulting in stronger preference alignment and superior performance on metrics like text consistency and expert evaluations.
Text-to-image generation has advanced rapidly, yet it still struggles to capture the nuanced user preferences. Existing approaches typically rely on multimodal large language models to infer user preferences, but the derived prompts or latent codes rarely reflect them faithfully, leading to suboptimal personalization. We present Premier, a novel preference modulation framework for personalized image generation. Premier represents each user's preference as a learnable embedding and introduces a preference adapter that fuses the user embedding with the text prompt. To enable accurate and fine-grained preference control, the fused preference embedding is further used to modulate the generative process. To enhance the distinctness of individual preference and improve alignment between outputs and user-specific styles, we incorporate a dispersion loss that enforces separation among user embeddings. When user data are scarce, new users are represented as linear combinations of existing preference embeddings learned during training, enabling effective generalization. Experiments show that Premier outperforms prior methods under the same history length, achieving stronger preference alignment and superior performance on text consistency, ViPer proxy metrics, and expert evaluations.