DiffBlender: Composable and Versatile Multimodal Text-to-Image Diffusion Models
This work addresses the need for more versatile and composable image synthesis tools for AI and creative applications, representing an incremental improvement by building on existing diffusion models with minimal parameter updates.
The paper tackles the problem of enhancing text-to-image generation by integrating multiple modalities like structure, layout, and attributes into a single diffusion model, achieving new benchmarks in multimodal generation through quantitative and qualitative comparisons.
In this study, we aim to enhance the capabilities of diffusion-based text-to-image (T2I) generation models by integrating diverse modalities beyond textual descriptions within a unified framework. To this end, we categorize widely used conditional inputs into three modality types: structure, layout, and attribute. We propose a multimodal T2I diffusion model, which is capable of processing all three modalities within a single architecture without modifying the parameters of the pre-trained diffusion model, as only a small subset of components is updated. Our approach sets new benchmarks in multimodal generation through extensive quantitative and qualitative comparisons with existing conditional generation methods. We demonstrate that DiffBlender effectively integrates multiple sources of information and supports diverse applications in detailed image synthesis. The code and demo are available at https://github.com/sungnyun/diffblender.