MagicFusion: Boosting Text-to-Image Generation Performance by Fusing Diffusion Models
This work addresses the challenge of improving text-to-image generation for AI and creative applications, but it is incremental as it builds on existing diffusion models without introducing a new paradigm.
The paper tackles the problem of combining strengths from multiple text-to-image diffusion models by proposing Saliency-aware Noise Blending (SNB), a training-free method that fuses models based on saliency to achieve more controllable generation, with extensive experiments demonstrating its effectiveness.
The advent of open-source AI communities has produced a cornucopia of powerful text-guided diffusion models that are trained on various datasets. While few explorations have been conducted on ensembling such models to combine their strengths. In this work, we propose a simple yet effective method called Saliency-aware Noise Blending (SNB) that can empower the fused text-guided diffusion models to achieve more controllable generation. Specifically, we experimentally find that the responses of classifier-free guidance are highly related to the saliency of generated images. Thus we propose to trust different models in their areas of expertise by blending the predicted noises of two diffusion models in a saliency-aware manner. SNB is training-free and can be completed within a DDIM sampling process. Additionally, it can automatically align the semantics of two noise spaces without requiring additional annotations such as masks. Extensive experiments show the impressive effectiveness of SNB in various applications. Project page is available at https://magicfusion.github.io/.