Learning to Customize Text-to-Image Diffusion In Diverse Context
This addresses a generalization issue in text-to-image customization for users needing adaptable models, though it is incremental as it builds on existing embedding-based methods.
The paper tackles the problem of text-to-image customization methods overfitting to training images and failing to generalize to new contexts, by diversifying context in textual space through rich prompts and self-supervised learning. This simple, cost-effective approach improves semantic alignment and prompt fidelity, achieving notable CLIP score improvements when combined with four baseline methods.
Most text-to-image customization techniques fine-tune models on a small set of \emph{personal concept} images captured in minimal contexts. This often results in the model becoming overfitted to these training images and unable to generalize to new contexts in future text prompts. Existing customization methods are built on the success of effectively representing personal concepts as textual embeddings. Thus, in this work, we resort to diversifying the context of these personal concepts \emph{solely} within the textual space by simply creating a contextually rich set of text prompts, together with a widely used self-supervised learning objective. Surprisingly, this straightforward and cost-effective method significantly improves semantic alignment in the textual space, and this effect further extends to the image space, resulting in higher prompt fidelity for generated images. Additionally, our approach does not require any architectural modifications, making it highly compatible with existing text-to-image customization methods. We demonstrate the broad applicability of our approach by combining it with four different baseline methods, achieving notable CLIP score improvements.