OmniPrism: Learning Disentangled Visual Concept for Image Generation
This addresses the challenge of generating creative images with multiple concepts without confusion, which is important for artists and designers, though it appears incremental as it builds on existing diffusion and disentanglement techniques.
The paper tackles the problem of concept confusion in multi-aspect creative image generation by proposing OmniPrism, a method that learns disentangled visual concepts from reference images using natural language guidance and a diffusion model, resulting in high-quality, concept-disentangled image generation with high fidelity to text prompts and desired concepts.
Creative visual concept generation often draws inspiration from specific concepts in a reference image to produce relevant outcomes. However, existing methods are typically constrained to single-aspect concept generation or are easily disrupted by irrelevant concepts in multi-aspect concept scenarios, leading to concept confusion and hindering creative generation. To address this, we propose OmniPrism, a visual concept disentangling approach for creative image generation. Our method learns disentangled concept representations guided by natural language and trains a diffusion model to incorporate these concepts. We utilize the rich semantic space of a multimodal extractor to achieve concept disentanglement from given images and concept guidance. To disentangle concepts with different semantics, we construct a paired concept disentangled dataset (PCD-200K), where each pair shares the same concept such as content, style, and composition. We learn disentangled concept representations through our contrastive orthogonal disentangled (COD) training pipeline, which are then injected into additional diffusion cross-attention layers for generation. A set of block embeddings is designed to adapt each block's concept domain in the diffusion models. Extensive experiments demonstrate that our method can generate high-quality, concept-disentangled results with high fidelity to text prompts and desired concepts.