Mitigating Semantic Collapse in Generative Personalization with Test-Time Embedding Adjustment
This addresses a critical issue for users of generative AI models by enhancing semantic richness in personalized image generation, though it is incremental as it builds on existing personalization methods.
The paper tackles the semantic collapse problem in generative personalization, where learned visual concepts dominate others in multi-concept prompts, leading to simplified outputs; they propose a training-free embedding adjustment method that significantly improves text-image alignment.
In this paper, we investigate the semantic collapsing problem in generative personalization, an under-explored topic where the learned visual concept ($V$) gradually shifts from its original textual meaning and comes to dominate other concepts in multi-concept input prompts. This issue not only reduces the semantic richness of complex input prompts like "a photo of $V$ wearing glasses and playing guitar" into simpler, less contextually rich forms such as "a photo of $V$" but also leads to simplified output images that fail to capture the intended concept. We identify the root cause as unconstrained optimisation, which allows the learned embedding $V$ to drift arbitrarily in the embedding space, both in direction and magnitude. To address this, we propose a simple yet effective training-free method that adjusts the magnitude and direction of pre-trained embedding at inference time, effectively mitigating the semantic collapsing problem. Our method is broadly applicable across different personalization methods and demonstrates significant improvements in text-image alignment in diverse use cases. Our code is anonymously published at https://github.com/tuananhbui89/Embedding-Adjustment