Piece it Together: Part-Based Concepting with IP-Priors
This addresses a need for visual designers who work with visual fragments beyond text, enabling creative exploration of concepts, but it appears incremental as it builds on existing IP-Adapter+ technology.
The authors tackled the problem of generating coherent visual concepts from partial user-provided components, such as wings or hairstyles, by introducing a framework that integrates these fragments and samples missing parts to produce plausible compositions. They achieved this by training IP-Prior, a lightweight flow-matching model on IP-Adapter+ representations, and improved prompt adherence through LoRA-based fine-tuning, though no concrete numbers are provided.
Advanced generative models excel at synthesizing images but often rely on text-based conditioning. Visual designers, however, often work beyond language, directly drawing inspiration from existing visual elements. In many cases, these elements represent only fragments of a potential concept-such as an uniquely structured wing, or a specific hairstyle-serving as inspiration for the artist to explore how they can come together creatively into a coherent whole. Recognizing this need, we introduce a generative framework that seamlessly integrates a partial set of user-provided visual components into a coherent composition while simultaneously sampling the missing parts needed to generate a plausible and complete concept. Our approach builds on a strong and underexplored representation space, extracted from IP-Adapter+, on which we train IP-Prior, a lightweight flow-matching model that synthesizes coherent compositions based on domain-specific priors, enabling diverse and context-aware generations. Additionally, we present a LoRA-based fine-tuning strategy that significantly improves prompt adherence in IP-Adapter+ for a given task, addressing its common trade-off between reconstruction quality and prompt adherence.